2023-12-27T21:31:17.552 INFO:root:teuthology version: 0.0.1.dev228+g01dd171 2023-12-27T21:31:17.552 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502301 --name yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502301/orig.config.yaml 2023-12-27T21:31:17.601 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-27T21:31:17.687 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502301 branch: wip-yuri7-testing-2023-12-27-1008-pacific description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: null first_in_suite: false job_id: '7502301' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri7-testing-2023-12-27-1008-pacific ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: c5564c7988cbaadc3382253af5843a8595347c2d ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c5564c7988cbaadc3382253af5843a8595347c2d selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri7-testing-2023-12-27-1008-pacific sha1: c5564c7988cbaadc3382253af5843a8595347c2d owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5573 sha1: c5564c7988cbaadc3382253af5843a8595347c2d sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri7-testing-2023-12-27-1008-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c5564c7988cbaadc3382253af5843a8595347c2d targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOblRzH4CISFZgKXd44gHxbtkO20DUUrc3Tk8ZWISfJl0F8R7ZnFxuu/tw3Bx18hMtCqyI+gKAQQNinSK5R6+80= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQEeKbf/5+MFgp1CU514Rt/z8u7LW2e6ayiIfhvGzRKJnbTd60HsQS9USBrvGz6/7ODtWja1GHRU4zoLdGn7aU= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 01dd171a561ca33631bf0a38b105879779f72410 timestamp: 2023-12-27_20:59:12 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-27T21:31:17.687 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa; will attempt to use it 2023-12-27T21:31:17.688 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa/tasks 2023-12-27T21:31:17.688 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-27T21:31:17.690 INFO:teuthology.task.internal:Checking packages... 2023-12-27T21:31:17.714 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'c5564c7988cbaadc3382253af5843a8595347c2d' 2023-12-27T21:31:17.714 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-27T21:31:17.714 INFO:teuthology.packaging:ref: None 2023-12-27T21:31:17.714 INFO:teuthology.packaging:tag: None 2023-12-27T21:31:17.714 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-12-27-1008-pacific 2023-12-27T21:31:17.714 INFO:teuthology.packaging:sha1: c5564c7988cbaadc3382253af5843a8595347c2d 2023-12-27T21:31:17.714 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2023-12-27-1008-pacific 2023-12-27T21:31:17.930 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-417.gc5564c79 2023-12-27T21:31:17.932 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-27T21:31:17.939 INFO:teuthology.task.internal:no buildpackages task found 2023-12-27T21:31:17.939 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-27T21:31:17.947 INFO:teuthology.task.internal:Saving configuration 2023-12-27T21:31:17.967 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-27T21:31:17.974 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-27T21:31:17.997 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-27 21:26:06.226454', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOblRzH4CISFZgKXd44gHxbtkO20DUUrc3Tk8ZWISfJl0F8R7ZnFxuu/tw3Bx18hMtCqyI+gKAQQNinSK5R6+80='} 2023-12-27T21:31:18.019 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-27 21:26:06.228371', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQEeKbf/5+MFgp1CU514Rt/z8u7LW2e6ayiIfhvGzRKJnbTd60HsQS9USBrvGz6/7ODtWja1GHRU4zoLdGn7aU='} 2023-12-27T21:31:18.019 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-27T21:31:18.027 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-27T21:31:18.027 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-27T21:31:18.027 INFO:teuthology.run_tasks:Running task console_log... 2023-12-27T21:31:18.103 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6b203a3e50>, signals=[15]) 2023-12-27T21:31:18.103 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-27T21:31:18.110 INFO:teuthology.task.internal:Opening connections... 2023-12-27T21:31:18.110 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2023-12-27T21:31:18.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-27T21:31:18.180 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2023-12-27T21:31:18.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-27T21:31:18.249 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-27T21:31:18.258 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2023-12-27T21:31:18.279 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2023-12-27T21:31:18.279 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Red Hat Enterprise Linux" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8.6 (Ootpa)" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:ID="rhel" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="fedora" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8.6" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.redhat.com/" 2023-12-27T21:31:18.338 INFO:teuthology.orchestra.run.smithi086.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-27T21:31:18.339 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-27T21:31:18.339 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:31:18.339 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-27T21:31:18.339 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-12-27T21:31:18.339 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-27T21:31:18.339 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-12-27T21:31:18.339 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-12-27T21:31:18.359 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2023-12-27T21:31:18.379 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2023-12-27T21:31:18.379 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2023-12-27T21:31:18.439 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Red Hat Enterprise Linux" 2023-12-27T21:31:18.492 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="8.6 (Ootpa)" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:ID="rhel" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="fedora" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="8.6" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el8" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.redhat.com/" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:31:18.493 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-27T21:31:18.494 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-12-27T21:31:18.494 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-27T21:31:18.494 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-12-27T21:31:18.494 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2023-12-27T21:31:18.516 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-27T21:31:18.528 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-27T21:31:18.539 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-27T21:31:18.539 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2023-12-27T21:31:18.542 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2023-12-27T21:31:18.561 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-27T21:31:18.569 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-27T21:31:18.570 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2023-12-27T21:31:18.601 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2023-12-27T21:31:18.626 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-27T21:31:18.684 INFO:teuthology.run_tasks:Running task kernel... 2023-12-27T21:31:18.703 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-27T21:31:18.703 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-27T21:31:18.704 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-27T21:31:18.704 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-27T21:31:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:31:18.723 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2023-12-27T21:31:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:31:18.724 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-12-27T21:31:18.781 INFO:teuthology.orchestra.run.smithi196.stdout:4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:18.782 INFO:teuthology.task.kernel:Running kernel on smithi196: 4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:18.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2023-12-27T21:31:18.784 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:18.784 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:18.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2023-12-27T21:31:20.271 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:31:20.271 INFO:teuthology.orchestra.run.smithi196.stdout:Unable to read consumer identity 2023-12-27T21:31:20.314 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:31:20.315 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2023-12-27T21:31:20.524 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:31:20.524 INFO:teuthology.orchestra.run.smithi196.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2023-12-27T21:31:20.524 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:31:20.740 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:31:20.740 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2023-12-27T21:31:20.741 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:31:22.303 INFO:teuthology.orchestra.run.smithi196.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-12-27T21:31:22.486 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-12-27T21:31:23.120 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-12-27T21:31:26.197 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2023-12-27T21:31:26.728 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 512 kB/s | 24 kB 00:00 2023-12-27T21:31:27.431 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2023-12-27T21:31:27.431 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-12-27T21:31:27.507 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:31:27.509 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:31:27.509 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:31:27.651 DEBUG:teuthology.orchestra.run.smithi196:> echo no | sudo yum reinstall kernel || true 2023-12-27T21:31:27.987 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:31:27.987 INFO:teuthology.orchestra.run.smithi196.stdout:Unable to read consumer identity 2023-12-27T21:31:28.266 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:31:28.266 INFO:teuthology.orchestra.run.smithi196.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2023-12-27T21:31:28.267 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:31:28.574 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:02 ago on Wed 27 Dec 2023 09:31:26 PM UTC. 2023-12-27T21:31:28.664 INFO:teuthology.orchestra.run.smithi196.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-27T21:31:28.665 INFO:teuthology.orchestra.run.smithi196.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-12-27T21:31:28.711 INFO:teuthology.orchestra.run.smithi196.stderr:Error: No packages marked for reinstall. 2023-12-27T21:31:28.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum reinstall -y kernel || true 2023-12-27T21:31:29.092 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:31:29.092 INFO:teuthology.orchestra.run.smithi196.stdout:Unable to read consumer identity 2023-12-27T21:31:29.355 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:31:29.355 INFO:teuthology.orchestra.run.smithi196.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2023-12-27T21:31:29.355 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:31:29.652 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:03 ago on Wed 27 Dec 2023 09:31:26 PM UTC. 2023-12-27T21:31:29.720 INFO:teuthology.orchestra.run.smithi196.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-27T21:31:29.720 INFO:teuthology.orchestra.run.smithi196.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-12-27T21:31:29.752 INFO:teuthology.orchestra.run.smithi196.stderr:Error: No packages marked for reinstall. 2023-12-27T21:31:29.795 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 649 kB/s | 24 kB 00:00 2023-12-27T21:31:29.798 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2023-12-27T21:31:29.970 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:29.970 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:29.971 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-12-27T21:31:29.971 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-27T21:31:29.971 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-27T21:31:30.004 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2023-12-27T21:31:30.027 DEBUG:teuthology.parallel:result is None 2023-12-27T21:31:30.497 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2023-12-27T21:31:30.497 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-12-27T21:31:30.581 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:31:30.582 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:31:30.582 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:31:30.737 DEBUG:teuthology.orchestra.run.smithi086:> echo no | sudo yum reinstall kernel || true 2023-12-27T21:31:31.074 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:31:31.075 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2023-12-27T21:31:31.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:31:31.343 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2023-12-27T21:31:31.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:31:31.648 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:02 ago on Wed 27 Dec 2023 09:31:29 PM UTC. 2023-12-27T21:31:31.731 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-27T21:31:31.739 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-12-27T21:31:31.772 INFO:teuthology.orchestra.run.smithi086.stderr:Error: No packages marked for reinstall. 2023-12-27T21:31:31.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum reinstall -y kernel || true 2023-12-27T21:31:32.152 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:31:32.152 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2023-12-27T21:31:32.458 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:31:32.458 INFO:teuthology.orchestra.run.smithi086.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2023-12-27T21:31:32.458 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:31:32.754 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:03 ago on Wed 27 Dec 2023 09:31:29 PM UTC. 2023-12-27T21:31:32.821 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-12-27T21:31:32.821 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-12-27T21:31:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error: No packages marked for reinstall. 2023-12-27T21:31:32.897 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-12-27T21:31:33.048 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:33.048 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2023-12-27T21:31:33.048 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-12-27T21:31:33.048 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-27T21:31:33.048 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-27T21:31:33.084 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2023-12-27T21:31:33.097 DEBUG:teuthology.parallel:result is None 2023-12-27T21:31:33.097 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-27T21:31:33.104 INFO:teuthology.task.internal:Creating test directory... 2023-12-27T21:31:33.104 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-27T21:31:33.140 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-27T21:31:33.160 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-27T21:31:33.167 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-27T21:31:33.172 INFO:teuthology.task.internal:Creating archive directory... 2023-12-27T21:31:33.172 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-27T21:31:33.198 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-27T21:31:33.243 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-27T21:31:33.250 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-27T21:31:33.251 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-27T21:31:33.262 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-27T21:31:33.289 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-27T21:31:33.297 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-27T21:31:33.315 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-27T21:31:33.327 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-27T21:31:33.328 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-27T21:31:33.335 INFO:teuthology.task.internal:Configuring sudo... 2023-12-27T21:31:33.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-27T21:31:33.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-27T21:31:33.398 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-27T21:31:33.407 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-27T21:31:33.407 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-27T21:31:33.409 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-27T21:31:33.457 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-27T21:31:33.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-27T21:31:33.568 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-27T21:31:33.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-27T21:31:33.653 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:31:33.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-27T21:31:33.723 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-27T21:31:33.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-27T21:31:33.819 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-27T21:31:33.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-27T21:31:33.949 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:31:33.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-27T21:31:34.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2023-12-27T21:31:34.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2023-12-27T21:31:34.059 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-27T21:31:34.120 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-27T21:31:34.549 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-27T21:31:34.557 INFO:teuthology.task.internal:Starting timer... 2023-12-27T21:31:34.557 INFO:teuthology.run_tasks:Running task pcp... 2023-12-27T21:31:34.567 INFO:teuthology.run_tasks:Running task selinux... 2023-12-27T21:31:34.576 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-27T21:31:34.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2023-12-27T21:31:34.635 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2023-12-27T21:31:34.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2023-12-27T21:31:34.703 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2023-12-27T21:31:34.705 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-27T21:31:34.705 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2023-12-27T21:31:34.726 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2023-12-27T21:31:34.726 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2023-12-27T21:31:34.754 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2023-12-27T21:31:34.754 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2023-12-27T21:31:34.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-27T21:31:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:31:34.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-27T21:31:34.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:31:34.824 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-27T21:31:34.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2023-12-27T21:31:34.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2023-12-27T21:31:34.899 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-27T21:31:34.909 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-27T21:31:34.915 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2023-12-27T21:31:35.081 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-27T21:31:35.097 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-27T21:31:35.098 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-27T21:39:10.410 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2023-12-27T21:39:10.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-27T21:39:10.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-27T21:39:10.480 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-12-27T21:39:10.547 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-12-27T21:39:10.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2023-12-27T21:39:10.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-27T21:39:10.617 DEBUG:teuthology.orchestra.run.smithi196:> true 2023-12-27T21:39:10.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2023-12-27T21:39:10.683 INFO:teuthology.run_tasks:Running task clock... 2023-12-27T21:39:10.694 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-27T21:39:10.694 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-27T21:39:10.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-27T21:39:10.697 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-27T21:39:10.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-27T21:39:10.727 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-27T21:39:10.742 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-27T21:39:10.755 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-27T21:39:10.769 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-27T21:39:10.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2023-12-27T21:39:10.823 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2023-12-27T21:39:10.838 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-27T21:39:10.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2023-12-27T21:39:10.852 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-27T21:39:10.855 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2023-12-27T21:39:10.869 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-27T21:39:10.883 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-27T21:39:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-12-27T21:39:10.934 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-27T21:39:10.935 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-12-27T21:39:10.935 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.935 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.935 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.935 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.958 INFO:teuthology.orchestra.run.smithi196.stderr:bash: ntpq: command not found 2023-12-27T21:39:10.962 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-27T21:39:10.962 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2023-12-27T21:39:10.962 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.962 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.962 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.962 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-27T21:39:10.963 INFO:teuthology.run_tasks:Running task install... 2023-12-27T21:39:10.972 DEBUG:teuthology.task.install:project ceph 2023-12-27T21:39:10.972 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}} 2023-12-27T21:39:10.972 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'} 2023-12-27T21:39:10.972 INFO:teuthology.task.install:Using flavor: default 2023-12-27T21:39:10.977 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-27T21:39:10.978 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-27T21:39:10.978 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d', 'tag': None, 'wait_for_package': False} 2023-12-27T21:39:10.978 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c5564c7988cbaadc3382253af5843a8595347c2d 2023-12-27T21:39:10.980 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d', 'tag': None, 'wait_for_package': False} 2023-12-27T21:39:10.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c5564c7988cbaadc3382253af5843a8595347c2d 2023-12-27T21:39:11.129 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/ 2023-12-27T21:39:11.129 INFO:teuthology.task.install.rpm:Package version is 16.2.14-417.gc5564c79 2023-12-27T21:39:11.137 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/ 2023-12-27T21:39:11.137 INFO:teuthology.task.install.rpm:Package version is 16.2.14-417.gc5564c79 2023-12-27T21:39:11.244 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-27T21:39:11.244 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:39:11.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-27T21:39:11.248 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-12-27-1008-pacific/c5564c7988cbaadc3382253af5843a8595347c2d/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-27T21:39:11.248 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:39:11.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-27T21:39:11.276 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-27T21:39:11.277 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c5564c7988cbaadc3382253af5843a8595347c2d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-27T21:39:11.278 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-27T21:39:11.278 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c5564c7988cbaadc3382253af5843a8595347c2d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-27T21:39:11.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-27T21:39:11.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-27T21:39:11.422 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-27T21:39:11.423 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-27T21:39:11.488 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2023-12-27T21:39:11.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2023-12-27T21:39:11.492 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2023-12-27T21:39:11.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2023-12-27T21:39:11.864 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:39:11.865 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:39:13.846 INFO:teuthology.orchestra.run.smithi086.stdout:52 files removed 2023-12-27T21:39:13.856 INFO:teuthology.orchestra.run.smithi196.stdout:52 files removed 2023-12-27T21:39:13.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2023-12-27T21:39:13.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2023-12-27T21:39:14.233 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:39:14.243 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:39:16.768 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 285 kB/s | 76 kB 00:00 2023-12-27T21:39:16.925 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 174 kB/s | 76 kB 00:00 2023-12-27T21:39:17.144 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 45 kB/s | 15 kB 00:00 2023-12-27T21:39:17.342 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 40 kB/s | 15 kB 00:00 2023-12-27T21:39:17.445 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 5.2 kB/s | 1.4 kB 00:00 2023-12-27T21:39:17.579 INFO:teuthology.orchestra.run.smithi196.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-12-27T21:39:17.718 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 4.1 kB/s | 1.4 kB 00:00 2023-12-27T21:39:17.877 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-12-27T21:39:18.938 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-12-27T21:39:19.971 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 16 MB 00:02 2023-12-27T21:39:22.503 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 429 kB/s | 24 kB 00:00 2023-12-27T21:39:23.238 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2023-12-27T21:39:23.480 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 372 kB/s | 24 kB 00:00 2023-12-27T21:39:24.881 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 64 MB 00:01 2023-12-27T21:39:26.089 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 58 MB 00:01 2023-12-27T21:39:35.640 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 58 MB 00:01 2023-12-27T21:39:36.989 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 MB/s | 64 MB 00:01 2023-12-27T21:39:46.223 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2023-12-27T21:39:51.932 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:39:51.935 INFO:teuthology.orchestra.run.smithi086.stdout:==================================================================================================== 2023-12-27T21:39:51.935 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:39:51.935 INFO:teuthology.orchestra.run.smithi086.stdout:==================================================================================================== 2023-12-27T21:39:51.935 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:39:51.935 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:16.2.14-417.gc5564c79.el8 ceph 12 M 2023-12-27T21:39:51.935 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 3.7 M 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 4.2 M 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:16.2.14-417.gc5564c79.el8 ceph 6.1 M 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:16.2.14-417.gc5564c79.el8 ceph 24 M 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:16.2.14-417.gc5564c79.el8 ceph 26 k 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 825 k 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-27T21:39:51.936 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 494 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 3.8 M 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:16.2.14-417.gc5564c79.el8 ceph 45 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:16.2.14-417.gc5564c79.el8 ceph 99 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:16.2.14-417.gc5564c79.el8 ceph 215 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:16.2.14-417.gc5564c79.el8 ceph 389 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:16.2.14-417.gc5564c79.el8 ceph 369 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:16.2.14-417.gc5564c79.el8 ceph 114 k 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:39:51.937 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:39:51.938 INFO:teuthology.orchestra.run.smithi086.stdout:==================================================================================================== 2023-12-27T21:39:51.938 INFO:teuthology.orchestra.run.smithi086.stdout:Install 21 Packages 2023-12-27T21:39:51.938 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2023-12-27T21:39:51.938 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:39:51.938 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 57 M 2023-12-27T21:39:51.938 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:39:52.840 INFO:teuthology.orchestra.run.smithi086.stdout:(1/23): ceph-base-16.2.14-417.gc5564c79.el8.x86 7.0 MB/s | 6.1 MB 00:00 2023-12-27T21:39:52.923 INFO:teuthology.orchestra.run.smithi086.stdout:(2/23): ceph-selinux-16.2.14-417.gc5564c79.el8. 308 kB/s | 26 kB 00:00 2023-12-27T21:39:53.040 INFO:teuthology.orchestra.run.smithi086.stdout:(3/23): libcephfs2-16.2.14-417.gc5564c79.el8.x8 6.9 MB/s | 825 kB 00:00 2023-12-27T21:39:53.132 INFO:teuthology.orchestra.run.smithi086.stdout:(4/23): libradosstriper1-16.2.14-417.gc5564c79. 5.3 MB/s | 494 kB 00:00 2023-12-27T21:39:53.348 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:39:53.351 INFO:teuthology.orchestra.run.smithi196.stdout:==================================================================================================== 2023-12-27T21:39:53.351 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:39:53.351 INFO:teuthology.orchestra.run.smithi196.stdout:==================================================================================================== 2023-12-27T21:39:53.351 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:39:53.351 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:16.2.14-417.gc5564c79.el8 ceph 12 M 2023-12-27T21:39:53.351 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 3.7 M 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 4.2 M 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:16.2.14-417.gc5564c79.el8 ceph 6.1 M 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:16.2.14-417.gc5564c79.el8 ceph 24 M 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:16.2.14-417.gc5564c79.el8 ceph 26 k 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 825 k 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 494 k 2023-12-27T21:39:53.352 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:16.2.14-417.gc5564c79.el8 ceph 3.8 M 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:16.2.14-417.gc5564c79.el8 ceph 45 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:16.2.14-417.gc5564c79.el8 ceph 99 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:16.2.14-417.gc5564c79.el8 ceph 215 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:16.2.14-417.gc5564c79.el8 ceph 389 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:16.2.14-417.gc5564c79.el8 ceph 369 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:16.2.14-417.gc5564c79.el8 ceph 114 k 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:==================================================================================================== 2023-12-27T21:39:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:Install 21 Packages 2023-12-27T21:39:53.354 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2023-12-27T21:39:53.354 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:39:53.354 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 57 M 2023-12-27T21:39:53.354 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:39:53.574 INFO:teuthology.orchestra.run.smithi086.stdout:(5/23): librgw2-16.2.14-417.gc5564c79.el8.x86_6 8.6 MB/s | 3.8 MB 00:00 2023-12-27T21:39:53.641 INFO:teuthology.orchestra.run.smithi086.stdout:(6/23): python3-ceph-argparse-16.2.14-417.gc556 678 kB/s | 45 kB 00:00 2023-12-27T21:39:53.716 INFO:teuthology.orchestra.run.smithi086.stdout:(7/23): python3-ceph-common-16.2.14-417.gc5564c 1.3 MB/s | 99 kB 00:00 2023-12-27T21:39:53.800 INFO:teuthology.orchestra.run.smithi086.stdout:(8/23): python3-cephfs-16.2.14-417.gc5564c79.el 2.5 MB/s | 215 kB 00:00 2023-12-27T21:39:53.917 INFO:teuthology.orchestra.run.smithi086.stdout:(9/23): python3-rados-16.2.14-417.gc5564c79.el8 3.3 MB/s | 389 kB 00:00 2023-12-27T21:39:54.009 INFO:teuthology.orchestra.run.smithi086.stdout:(10/23): python3-rbd-16.2.14-417.gc5564c79.el8. 3.9 MB/s | 369 kB 00:00 2023-12-27T21:39:54.234 INFO:teuthology.orchestra.run.smithi086.stdout:(11/23): ceph-common-16.2.14-417.gc5564c79.el8. 11 MB/s | 24 MB 00:02 2023-12-27T21:39:54.259 INFO:teuthology.orchestra.run.smithi086.stdout:(12/23): python3-rgw-16.2.14-417.gc5564c79.el8. 457 kB/s | 114 kB 00:00 2023-12-27T21:39:54.635 INFO:teuthology.orchestra.run.smithi086.stdout:(13/23): ceph-radosgw-16.2.14-417.gc5564c79.el8 4.4 MB/s | 12 MB 00:02 2023-12-27T21:39:54.726 INFO:teuthology.orchestra.run.smithi086.stdout:(14/23): leveldb-1.22-1.el8.x86_64.rpm 387 kB/s | 181 kB 00:00 2023-12-27T21:39:54.802 INFO:teuthology.orchestra.run.smithi086.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 540 kB/s | 306 kB 00:00 2023-12-27T21:39:54.852 INFO:teuthology.orchestra.run.smithi086.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 600 kB/s | 75 kB 00:00 2023-12-27T21:39:54.977 INFO:teuthology.orchestra.run.smithi086.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 251 kB/s | 44 kB 00:00 2023-12-27T21:39:55.002 INFO:teuthology.orchestra.run.smithi086.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 160 kB/s | 59 kB 00:00 2023-12-27T21:39:55.028 INFO:teuthology.orchestra.run.smithi086.stdout:(19/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-12-27T21:39:55.148 INFO:teuthology.orchestra.run.smithi196.stdout:(1/23): ceph-base-16.2.14-417.gc5564c79.el8.x86 4.1 MB/s | 6.1 MB 00:01 2023-12-27T21:39:55.153 INFO:teuthology.orchestra.run.smithi086.stdout:(20/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 313 kB/s | 47 kB 00:00 2023-12-27T21:39:55.187 INFO:teuthology.orchestra.run.smithi086.stdout:(21/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.7 MB/s | 354 kB 00:00 2023-12-27T21:39:55.215 INFO:teuthology.orchestra.run.smithi196.stdout:(2/23): ceph-selinux-16.2.14-417.gc5564c79.el8. 386 kB/s | 26 kB 00:00 2023-12-27T21:39:55.445 INFO:teuthology.orchestra.run.smithi086.stdout:(22/23): librados2-16.2.14-417.gc5564c79.el8.x8 9.1 MB/s | 3.7 MB 00:00 2023-12-27T21:39:55.623 INFO:teuthology.orchestra.run.smithi196.stdout:(3/23): libcephfs2-16.2.14-417.gc5564c79.el8.x8 2.0 MB/s | 825 kB 00:00 2023-12-27T21:39:55.766 INFO:teuthology.orchestra.run.smithi196.stdout:(4/23): libradosstriper1-16.2.14-417.gc5564c79. 3.4 MB/s | 494 kB 00:00 2023-12-27T21:39:55.919 INFO:teuthology.orchestra.run.smithi086.stdout:(23/23): librbd1-16.2.14-417.gc5564c79.el8.x86_ 5.5 MB/s | 4.2 MB 00:00 2023-12-27T21:39:55.921 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:39:55.921 INFO:teuthology.orchestra.run.smithi086.stdout:Total 14 MB/s | 57 MB 00:03 2023-12-27T21:39:55.972 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:39:56.018 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:39:56.291 INFO:teuthology.orchestra.run.smithi196.stdout:(5/23): ceph-radosgw-16.2.14-417.gc5564c79.el8. 4.5 MB/s | 12 MB 00:02 2023-12-27T21:39:56.338 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:39:56.347 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:39:56.349 INFO:teuthology.orchestra.run.smithi196.stdout:(6/23): python3-ceph-argparse-16.2.14-417.gc556 774 kB/s | 45 kB 00:00 2023-12-27T21:39:56.425 INFO:teuthology.orchestra.run.smithi196.stdout:(7/23): librgw2-16.2.14-417.gc5564c79.el8.x86_6 5.8 MB/s | 3.8 MB 00:00 2023-12-27T21:39:56.450 INFO:teuthology.orchestra.run.smithi196.stdout:(8/23): python3-ceph-common-16.2.14-417.gc5564c 986 kB/s | 99 kB 00:00 2023-12-27T21:39:56.517 INFO:teuthology.orchestra.run.smithi196.stdout:(9/23): python3-cephfs-16.2.14-417.gc5564c79.el 2.3 MB/s | 215 kB 00:00 2023-12-27T21:39:56.542 INFO:teuthology.orchestra.run.smithi196.stdout:(10/23): python3-rados-16.2.14-417.gc5564c79.el 4.1 MB/s | 389 kB 00:00 2023-12-27T21:39:56.593 INFO:teuthology.orchestra.run.smithi196.stdout:(11/23): python3-rgw-16.2.14-417.gc5564c79.el8. 2.2 MB/s | 114 kB 00:00 2023-12-27T21:39:56.843 INFO:teuthology.orchestra.run.smithi196.stdout:(12/23): ceph-common-16.2.14-417.gc5564c79.el8. 7.7 MB/s | 24 MB 00:03 2023-12-27T21:39:56.868 INFO:teuthology.orchestra.run.smithi196.stdout:(13/23): python3-rbd-16.2.14-417.gc5564c79.el8. 1.0 MB/s | 369 kB 00:00 2023-12-27T21:39:56.985 INFO:teuthology.orchestra.run.smithi196.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 505 kB/s | 59 kB 00:00 2023-12-27T21:39:57.010 INFO:teuthology.orchestra.run.smithi196.stdout:(15/23): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2023-12-27T21:39:57.027 INFO:teuthology.orchestra.run.smithi196.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 705 kB/s | 306 kB 00:00 2023-12-27T21:39:57.052 INFO:teuthology.orchestra.run.smithi196.stdout:(17/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-12-27T21:39:57.161 INFO:teuthology.orchestra.run.smithi196.stdout:(18/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-12-27T21:39:57.212 INFO:teuthology.orchestra.run.smithi196.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 239 kB/s | 44 kB 00:00 2023-12-27T21:39:57.261 INFO:teuthology.orchestra.run.smithi196.stdout:(20/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2023-12-27T21:39:57.394 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:39:57.397 INFO:teuthology.orchestra.run.smithi196.stdout:(21/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2023-12-27T21:39:57.584 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-27T21:39:57.704 INFO:teuthology.orchestra.run.smithi196.stdout:(22/23): librbd1-16.2.14-417.gc5564c79.el8.x86_ 9.5 MB/s | 4.2 MB 00:00 2023-12-27T21:39:57.829 INFO:teuthology.orchestra.run.smithi196.stdout:(23/23): librados2-16.2.14-417.gc5564c79.el8.x8 6.1 MB/s | 3.7 MB 00:00 2023-12-27T21:39:57.831 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:39:57.831 INFO:teuthology.orchestra.run.smithi196.stdout:Total 13 MB/s | 57 MB 00:04 2023-12-27T21:39:57.880 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:39:57.927 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:39:57.928 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:39:57.967 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-27T21:39:58.176 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:16.2.14-417.gc5564c79.el8.x86_64 2/25 2023-12-27T21:39:58.252 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:39:58.259 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:39:58.277 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.14-417.gc5564c79.el8.x86_64 2/25 2023-12-27T21:39:58.473 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 3/25 2023-12-27T21:39:58.707 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-12-27T21:39:58.935 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 5/25 2023-12-27T21:39:59.276 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:39:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 5/25 2023-12-27T21:39:59.410 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-27T21:39:59.559 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 6/25 2023-12-27T21:39:59.625 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 6/25 2023-12-27T21:39:59.793 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-12-27T21:39:59.849 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-12-27T21:40:00.018 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-12-27T21:40:00.027 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:16.2.14-417.gc5564c79.el8.x86_64 2/25 2023-12-27T21:40:00.127 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.14-417.gc5564c79.el8.x86_64 2/25 2023-12-27T21:40:00.307 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 3/25 2023-12-27T21:40:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-12-27T21:40:00.541 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-12-27T21:40:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 9/25 2023-12-27T21:40:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 9/25 2023-12-27T21:40:00.727 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 5/25 2023-12-27T21:40:00.881 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-27T21:40:01.048 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:16.2.14-417.gc5564c79.el8. 11/25 2023-12-27T21:40:01.128 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 5/25 2023-12-27T21:40:01.229 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 12/25 2023-12-27T21:40:01.485 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 6/25 2023-12-27T21:40:01.560 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 13/25 2023-12-27T21:40:01.578 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 6/25 2023-12-27T21:40:01.735 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/25 2023-12-27T21:40:01.800 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 14/25 2023-12-27T21:40:01.803 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/25 2023-12-27T21:40:02.017 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_6 15/25 2023-12-27T21:40:02.094 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_6 15/25 2023-12-27T21:40:02.243 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-27T21:40:02.264 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-12-27T21:40:02.435 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 9/25 2023-12-27T21:40:02.484 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-27T21:40:02.517 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 9/25 2023-12-27T21:40:02.740 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-27T21:40:02.755 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-27T21:40:02.882 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:16.2.14-417.gc5564c79.el8. 11/25 2023-12-27T21:40:02.964 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:16.2.14-417.gc5564c79.el8.x8 19/25 2023-12-27T21:40:03.063 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 12/25 2023-12-27T21:40:03.302 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 13/25 2023-12-27T21:40:03.526 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 14/25 2023-12-27T21:40:03.693 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_6 15/25 2023-12-27T21:40:03.763 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_6 15/25 2023-12-27T21:40:03.910 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-27T21:40:04.101 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-27T21:40:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-27T21:40:04.549 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:16.2.14-417.gc5564c79.el8.x8 19/25 2023-12-27T21:40:06.774 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 20/25 2023-12-27T21:40:07.042 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 20/25 2023-12-27T21:40:07.719 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 20/25 2023-12-27T21:40:08.043 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 21/25 2023-12-27T21:40:08.128 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 21/25 2023-12-27T21:40:08.342 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 22/25 2023-12-27T21:40:08.516 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 20/25 2023-12-27T21:40:08.743 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 20/25 2023-12-27T21:40:09.424 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 20/25 2023-12-27T21:40:09.777 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 21/25 2023-12-27T21:40:09.847 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 21/25 2023-12-27T21:40:10.018 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 22/25 2023-12-27T21:40:27.190 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 22/25 2023-12-27T21:40:27.190 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2023-12-27T21:40:27.190 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2023-12-27T21:40:27.190 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-12-27T21:40:27.191 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2023-12-27T21:40:27.191 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2023-12-27T21:40:27.191 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2023-12-27T21:40:27.191 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2023-12-27T21:40:27.191 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2023-12-27T21:40:27.191 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-12-27T21:40:27.191 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 23/25 2023-12-27T21:40:28.462 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 23/25 2023-12-27T21:40:28.462 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-27T21:40:28.716 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-27T21:40:28.717 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-27T21:40:29.071 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 22/25 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /dev 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /run 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /dev 2023-12-27T21:40:29.072 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:40:29.385 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-27T21:40:29.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 1/25 2023-12-27T21:40:29.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 2/25 2023-12-27T21:40:29.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 3/25 2023-12-27T21:40:29.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 4/25 2023-12-27T21:40:29.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 5/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_6 6/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 7/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:16.2.14-417.gc5564c79.el8. 8/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:16.2.14-417.gc5564c79.el8.x8 9/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 10/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 11/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 12/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 13/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-12-27T21:40:29.386 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 20/25 2023-12-27T21:40:29.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 21/25 2023-12-27T21:40:29.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.14-417.gc5564c79.el8.x86_64 22/25 2023-12-27T21:40:29.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-27T21:40:29.387 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 24/25 2023-12-27T21:40:30.256 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 23/25 2023-12-27T21:40:30.331 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 23/25 2023-12-27T21:40:30.331 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-27T21:40:30.565 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-27T21:40:30.566 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 1/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 2/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 3/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 4/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 5/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_6 6/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 7/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:16.2.14-417.gc5564c79.el8. 8/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:16.2.14-417.gc5564c79.el8.x8 9/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 10/25 2023-12-27T21:40:31.217 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 11/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 12/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 13/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 18/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.14-417.gc5564c79.el8.x86_64 22/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-27T21:40:31.218 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 24/25 2023-12-27T21:40:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-27T21:40:45.774 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:40:46.302 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:40:46.302 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: leveldb-1.22-1.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-27T21:40:46.303 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:40:46.304 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:40:46.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2023-12-27T21:40:46.881 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:40:47.640 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-27T21:40:47.641 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.165 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: leveldb-1.22-1.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-27T21:40:48.166 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:40:48.167 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:40:48.167 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:40:48.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2023-12-27T21:40:48.722 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:40:49.010 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2023-12-27T21:40:49.600 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:40:51.304 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:40:51.630 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2023-12-27T21:40:51.813 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:40:51.813 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2023-12-27T21:40:51.813 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:16.2.14-417.gc5564c79.el8 ceph 52 M 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:16.2.14-417.gc5564c79.el8 ceph 174 k 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2023-12-27T21:40:51.814 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-12-27T21:40:51.815 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:40:51.815 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 53 M 2023-12-27T21:40:51.815 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 196 M 2023-12-27T21:40:51.815 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:40:51.849 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 70 kB 00:00 2023-12-27T21:40:52.016 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): libcephsqlite-16.2.14-417.gc5564c79.el8. 868 kB/s | 174 kB 00:00 2023-12-27T21:40:52.041 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-12-27T21:40:52.167 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2023-12-27T21:40:52.209 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2023-12-27T21:40:54.126 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:16.2.14-417.gc5564c79.el8 ceph 52 M 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-12-27T21:40:54.127 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:16.2.14-417.gc5564c79.el8 ceph 174 k 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2023-12-27T21:40:54.128 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:40:54.129 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 53 M 2023-12-27T21:40:54.129 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 196 M 2023-12-27T21:40:54.129 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:40:54.160 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.2 MB/s | 70 kB 00:00 2023-12-27T21:40:54.344 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): libcephsqlite-16.2.14-417.gc5564c79.el8. 808 kB/s | 174 kB 00:00 2023-12-27T21:40:54.369 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 936 kB/s | 187 kB 00:00 2023-12-27T21:40:54.400 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-16.2.14-417.gc5564c79.el8.x86_ 20 MB/s | 52 MB 00:02 2023-12-27T21:40:54.402 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:40:54.403 INFO:teuthology.orchestra.run.smithi086.stdout:Total 20 MB/s | 53 MB 00:02 2023-12-27T21:40:54.428 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-12-27T21:40:54.441 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:40:54.462 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:40:54.462 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:40:54.520 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): jq-1.6-7.el8.x86_64.rpm 1.4 MB/s | 202 kB 00:00 2023-12-27T21:40:54.758 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:40:54.767 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:40:55.296 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:40:55.455 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 1/6 2023-12-27T21:40:55.585 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-27T21:40:55.704 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-27T21:40:55.895 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-7.el8.x86_64 3/6 2023-12-27T21:40:56.075 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-12-27T21:40:56.287 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 5/6 2023-12-27T21:40:57.262 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-16.2.14-417.gc5564c79.el8.x86_ 17 MB/s | 52 MB 00:03 2023-12-27T21:40:57.263 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:40:57.264 INFO:teuthology.orchestra.run.smithi196.stdout:Total 17 MB/s | 53 MB 00:03 2023-12-27T21:40:57.303 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:40:57.324 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:40:57.324 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:40:57.618 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:40:57.627 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:40:58.197 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:40:58.361 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 1/6 2023-12-27T21:40:58.516 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-27T21:40:58.598 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-12-27T21:40:58.750 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-7.el8.x86_64 3/6 2023-12-27T21:40:58.897 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-12-27T21:40:59.098 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 5/6 2023-12-27T21:41:00.842 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 5/6 2023-12-27T21:41:01.091 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 6/6 2023-12-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 6/6 2023-12-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 1/6 2023-12-27T21:41:01.350 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 2/6 2023-12-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-12-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-27T21:41:01.351 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-27T21:41:03.658 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 5/6 2023-12-27T21:41:03.897 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 6/6 2023-12-27T21:41:04.146 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 6/6 2023-12-27T21:41:04.146 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 1/6 2023-12-27T21:41:04.146 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 2/6 2023-12-27T21:41:04.146 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-12-27T21:41:04.147 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-27T21:41:04.147 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-27T21:41:19.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-7.el8.x86_64 6/6 2023-12-27T21:41:19.767 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:41:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:41:20.304 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:41:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:20.304 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-7.el8.x86_64 2023-12-27T21:41:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-27T21:41:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-27T21:41:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-27T21:41:20.305 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:41:20.305 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:41:20.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2023-12-27T21:41:20.778 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:41:23.361 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-7.el8.x86_64 6/6 2023-12-27T21:41:23.361 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:41:23.729 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2023-12-27T21:41:23.884 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:41:23.884 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:41:23.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:23.884 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-7.el8.x86_64 2023-12-27T21:41:23.885 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:23.885 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-27T21:41:23.885 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-27T21:41:23.885 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-27T21:41:23.885 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:41:23.885 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:41:24.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2023-12-27T21:41:24.379 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:41:24.433 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 17 kB/s | 4.1 kB 00:00 2023-12-27T21:41:26.713 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================= 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================= 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:16.2.14-417.gc5564c79.el8 ceph 6.4 k 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:16.2.14-417.gc5564c79.el8 ceph 2.4 M 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:16.2.14-417.gc5564c79.el8 ceph 1.6 M 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 245 k 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:16.2.14-417.gc5564c79.el8 ceph 5.0 M 2023-12-27T21:41:26.716 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:16.2.14-417.gc5564c79.el8 ceph 18 M 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-27T21:41:26.717 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-27T21:41:26.718 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-27T21:41:26.719 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-12-27T21:41:26.719 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-27T21:41:26.719 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:41:26.719 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:41:26.719 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================= 2023-12-27T21:41:26.719 INFO:teuthology.orchestra.run.smithi086.stdout:Install 32 Packages 2023-12-27T21:41:26.719 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:41:26.720 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 M 2023-12-27T21:41:26.720 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 101 M 2023-12-27T21:41:26.720 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:41:27.036 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-16.2.14-417.gc5564c79.el8.x86_64.r 36 kB/s | 6.4 kB 00:00 2023-12-27T21:41:27.186 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-mgr-16.2.14-417.gc5564c79.el8.x86_ 5.0 MB/s | 1.6 MB 00:00 2023-12-27T21:41:27.200 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2023-12-27T21:41:27.228 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): ceph-mds-16.2.14-417.gc5564c79.el8.x86_ 6.6 MB/s | 2.4 MB 00:00 2023-12-27T21:41:27.312 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): ceph-mgr-modules-core-16.2.14-417.gc556 2.9 MB/s | 245 kB 00:00 2023-12-27T21:41:27.404 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 475 kB/s | 44 kB 00:00 2023-12-27T21:41:27.479 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 2.4 MB/s | 185 kB 00:00 2023-12-27T21:41:27.521 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 4.0 MB/s | 173 kB 00:00 2023-12-27T21:41:27.570 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2023-12-27T21:41:27.579 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 6.5 MB/s | 384 kB 00:00 2023-12-27T21:41:27.622 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 266 kB/s | 11 kB 00:00 2023-12-27T21:41:27.655 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 528 kB/s | 18 kB 00:00 2023-12-27T21:41:27.689 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2023-12-27T21:41:27.722 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2023-12-27T21:41:27.764 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): python3-pecan-1.3.2-9.el8.noarch.rpm 6.6 MB/s | 283 kB 00:00 2023-12-27T21:41:27.798 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 468 kB/s | 16 kB 00:00 2023-12-27T21:41:27.832 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): python3-simplegeneric-0.8.1-17.el8.noa 559 kB/s | 19 kB 00:00 2023-12-27T21:41:27.890 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): python3-singledispatch-3.4.0.3-18.el8. 411 kB/s | 24 kB 00:00 2023-12-27T21:41:27.924 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): python3-tempora-1.14.1-5.el8.noarch.rp 878 kB/s | 29 kB 00:00 2023-12-27T21:41:28.116 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): ceph-osd-16.2.14-417.gc5564c79.el8.x86 20 MB/s | 18 MB 00:00 2023-12-27T21:41:28.141 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 127 kB/s | 27 kB 00:00 2023-12-27T21:41:28.183 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.6 MB/s | 251 kB 00:00 2023-12-27T21:41:28.217 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.5 MB/s | 85 kB 00:00 2023-12-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): python3-zc-lockfile-2.0-2.el8.noarch.r 691 kB/s | 23 kB 00:00 2023-12-27T21:41:28.351 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): ceph-mon-16.2.14-417.gc5564c79.el8.x86 3.8 MB/s | 5.0 MB 00:01 2023-12-27T21:41:28.409 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): python3-waitress-1.4.3-1.el8.noarch.rp 872 kB/s | 255 kB 00:00 2023-12-27T21:41:28.451 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2023-12-27T21:41:28.552 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): python3-markupsafe-0.23-19.el8.x86_64. 199 kB/s | 39 kB 00:00 2023-12-27T21:41:28.577 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 641 kB/s | 103 kB 00:00 2023-12-27T21:41:28.619 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): python3-pytz-2017.2-11.el8.noarch.rpm 812 kB/s | 54 kB 00:00 2023-12-27T21:41:28.644 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): python3-mako-1.0.6-14.el8.noarch.rpm 839 kB/s | 157 kB 00:00 2023-12-27T21:41:28.793 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-12-27T21:41:28.820 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): python3-pycparser-2.14-14.el8.noarch.r 450 kB/s | 109 kB 00:00 2023-12-27T21:41:28.845 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): python3-cryptography-3.2.1-6.el8.x86_6 2.7 MB/s | 559 kB 00:00 2023-12-27T21:41:28.848 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:41:28.849 INFO:teuthology.orchestra.run.smithi086.stdout:Total 15 MB/s | 31 MB 00:02 2023-12-27T21:41:28.941 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:41:28.978 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:41:28.978 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:41:29.203 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:41:29.212 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:41:30.056 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================= 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================= 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:16.2.14-417.gc5564c79.el8 ceph 6.4 k 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:16.2.14-417.gc5564c79.el8 ceph 2.4 M 2023-12-27T21:41:30.059 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:16.2.14-417.gc5564c79.el8 ceph 1.6 M 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 245 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:16.2.14-417.gc5564c79.el8 ceph 5.0 M 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:16.2.14-417.gc5564c79.el8 ceph 18 M 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-12-27T21:41:30.060 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-27T21:41:30.061 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================= 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout:Install 32 Packages 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 M 2023-12-27T21:41:30.062 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 101 M 2023-12-27T21:41:30.063 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:41:30.327 INFO:teuthology.orchestra.run.smithi196.stdout:(1/32): ceph-16.2.14-417.gc5564c79.el8.x86_64.r 51 kB/s | 6.4 kB 00:00 2023-12-27T21:41:30.553 INFO:teuthology.orchestra.run.smithi196.stdout:(2/32): ceph-mds-16.2.14-417.gc5564c79.el8.x86_ 7.0 MB/s | 2.4 MB 00:00 2023-12-27T21:41:30.586 INFO:teuthology.orchestra.run.smithi196.stdout:(3/32): ceph-mgr-16.2.14-417.gc5564c79.el8.x86_ 4.2 MB/s | 1.6 MB 00:00 2023-12-27T21:41:31.020 INFO:teuthology.orchestra.run.smithi196.stdout:(4/32): ceph-mgr-modules-core-16.2.14-417.gc556 564 kB/s | 245 kB 00:00 2023-12-27T21:41:31.433 INFO:teuthology.orchestra.run.smithi196.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 106 kB/s | 44 kB 00:00 2023-12-27T21:41:31.612 INFO:teuthology.orchestra.run.smithi196.stdout:(6/32): ceph-mon-16.2.14-417.gc5564c79.el8.x86_ 3.9 MB/s | 5.0 MB 00:01 2023-12-27T21:41:31.796 INFO:teuthology.orchestra.run.smithi196.stdout:(7/32): ceph-osd-16.2.14-417.gc5564c79.el8.x86_ 15 MB/s | 18 MB 00:01 2023-12-27T21:41:31.821 INFO:teuthology.orchestra.run.smithi196.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 477 kB/s | 185 kB 00:00 2023-12-27T21:41:31.963 INFO:teuthology.orchestra.run.smithi196.stdout:(9/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 2.3 MB/s | 384 kB 00:00 2023-12-27T21:41:31.988 INFO:teuthology.orchestra.run.smithi196.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 66 kB/s | 11 kB 00:00 2023-12-27T21:41:32.040 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:41:32.072 INFO:teuthology.orchestra.run.smithi196.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2023-12-27T21:41:32.114 INFO:teuthology.orchestra.run.smithi196.stdout:(12/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 344 kB/s | 173 kB 00:00 2023-12-27T21:41:32.156 INFO:teuthology.orchestra.run.smithi196.stdout:(13/32): python3-logutils-0.3.5-11.el8.noarch.r 291 kB/s | 49 kB 00:00 2023-12-27T21:41:32.181 INFO:teuthology.orchestra.run.smithi196.stdout:(14/32): python3-more-itertools-7.2.0-3.el8.noa 540 kB/s | 59 kB 00:00 2023-12-27T21:41:32.256 INFO:teuthology.orchestra.run.smithi196.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-12-27T21:41:32.281 INFO:teuthology.orchestra.run.smithi196.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-12-27T21:41:32.286 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-27T21:41:32.307 INFO:teuthology.orchestra.run.smithi196.stdout:(17/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-12-27T21:41:32.357 INFO:teuthology.orchestra.run.smithi196.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2023-12-27T21:41:32.382 INFO:teuthology.orchestra.run.smithi196.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2023-12-27T21:41:32.408 INFO:teuthology.orchestra.run.smithi196.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 273 kB/s | 27 kB 00:00 2023-12-27T21:41:32.483 INFO:teuthology.orchestra.run.smithi196.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2023-12-27T21:41:32.500 INFO:teuthology.orchestra.run.smithi196.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 930 kB/s | 85 kB 00:00 2023-12-27T21:41:32.535 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-27T21:41:32.584 INFO:teuthology.orchestra.run.smithi196.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 228 kB/s | 23 kB 00:00 2023-12-27T21:41:32.631 INFO:teuthology.orchestra.run.smithi196.stdout:(24/32): python3-waitress-1.4.3-1.el8.noarch.rp 934 kB/s | 255 kB 00:00 2023-12-27T21:41:32.751 INFO:teuthology.orchestra.run.smithi196.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 435 kB/s | 109 kB 00:00 2023-12-27T21:41:32.767 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2023-12-27T21:41:32.793 INFO:teuthology.orchestra.run.smithi196.stdout:(26/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-12-27T21:41:32.818 INFO:teuthology.orchestra.run.smithi196.stdout:(27/32): python3-cryptography-3.2.1-6.el8.x86_6 3.0 MB/s | 559 kB 00:00 2023-12-27T21:41:32.868 INFO:teuthology.orchestra.run.smithi196.stdout:(28/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2023-12-27T21:41:32.927 INFO:teuthology.orchestra.run.smithi196.stdout:(29/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 954 kB/s | 103 kB 00:00 2023-12-27T21:41:32.944 INFO:teuthology.orchestra.run.smithi196.stdout:(30/32): python3-mako-1.0.6-14.el8.noarch.rpm 2.0 MB/s | 157 kB 00:00 2023-12-27T21:41:32.986 INFO:teuthology.orchestra.run.smithi196.stdout:(31/32): python3-markupsafe-0.23-19.el8.x86_64. 201 kB/s | 39 kB 00:00 2023-12-27T21:41:33.153 INFO:teuthology.orchestra.run.smithi196.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 241 kB/s | 54 kB 00:00 2023-12-27T21:41:33.155 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:41:33.155 INFO:teuthology.orchestra.run.smithi196.stdout:Total 10 MB/s | 31 MB 00:03 2023-12-27T21:41:33.189 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 4/32 2023-12-27T21:41:33.254 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:41:33.292 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:41:33.292 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:41:33.425 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 5/32 2023-12-27T21:41:33.518 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:41:33.526 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:41:33.622 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2023-12-27T21:41:33.807 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2023-12-27T21:41:34.049 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2023-12-27T21:41:34.219 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 9/32 2023-12-27T21:41:34.464 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 10/32 2023-12-27T21:41:34.688 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:41:34.762 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 11/32 2023-12-27T21:41:34.906 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-27T21:41:35.025 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 12/32 2023-12-27T21:41:35.139 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-27T21:41:35.246 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-12-27T21:41:35.287 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2023-12-27T21:41:35.495 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-12-27T21:41:35.498 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-12-27T21:41:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-12-27T21:41:35.801 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2023-12-27T21:41:35.859 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-12-27T21:41:36.012 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2023-12-27T21:41:36.017 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-12-27T21:41:36.144 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-12-27T21:41:36.244 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-12-27T21:41:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-12-27T21:41:36.575 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-12-27T21:41:36.639 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2023-12-27T21:41:36.752 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-27T21:41:36.992 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-27T21:41:37.260 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-27T21:41:37.489 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-12-27T21:41:37.778 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-12-27T21:41:38.000 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 9/32 2023-12-27T21:41:38.118 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-27T21:41:38.184 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-12-27T21:41:38.336 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-12-27T21:41:38.442 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-417.gc5564c79.el8. 27/32 2023-12-27T21:41:38.559 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-12-27T21:41:38.614 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 28/32 2023-12-27T21:41:38.773 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-12-27T21:41:38.964 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-12-27T21:41:39.179 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-12-27T21:41:39.353 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-12-27T21:41:39.528 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-12-27T21:41:39.697 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-12-27T21:41:39.904 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-12-27T21:41:40.086 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-12-27T21:41:40.104 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 28/32 2023-12-27T21:41:40.289 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-27T21:41:40.338 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 29/32 2023-12-27T21:41:40.520 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-27T21:41:40.748 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-27T21:41:40.765 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 29/32 2023-12-27T21:41:40.930 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 30/32 2023-12-27T21:41:41.019 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-12-27T21:41:41.149 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 30/32 2023-12-27T21:41:41.292 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-12-27T21:41:41.296 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 31/32 2023-12-27T21:41:41.315 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 31/32 2023-12-27T21:41:41.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:16.2.14-417.gc5564c79.el8.x86_64 32/32 2023-12-27T21:41:41.638 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-27T21:41:41.845 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:16.2.14-417.gc5564c79.el8.x86_64 32/32 2023-12-27T21:41:41.845 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:16.2.14-417.gc5564c79.el8.x86_64 1/32 2023-12-27T21:41:41.845 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 2/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 3/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 4/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 5/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-417.gc5564c79.el8. 6/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-12-27T21:41:41.846 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/32 2023-12-27T21:41:41.847 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/32 2023-12-27T21:41:41.848 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2023-12-27T21:41:41.848 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 28/32 2023-12-27T21:41:41.848 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 29/32 2023-12-27T21:41:41.848 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 30/32 2023-12-27T21:41:41.848 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 31/32 2023-12-27T21:41:41.979 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-417.gc5564c79.el8. 27/32 2023-12-27T21:41:42.126 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 28/32 2023-12-27T21:41:43.613 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 28/32 2023-12-27T21:41:43.824 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 29/32 2023-12-27T21:41:44.252 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 29/32 2023-12-27T21:41:44.458 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 30/32 2023-12-27T21:41:44.675 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 30/32 2023-12-27T21:41:44.866 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 31/32 2023-12-27T21:41:44.884 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 31/32 2023-12-27T21:41:44.998 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:16.2.14-417.gc5564c79.el8.x86_64 32/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:16.2.14-417.gc5564c79.el8.x86_64 32/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:16.2.14-417.gc5564c79.el8.x86_64 1/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 2/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 3/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 4/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 5/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-417.gc5564c79.el8. 6/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-12-27T21:41:45.399 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-12-27T21:41:45.400 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-12-27T21:41:45.401 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-12-27T21:41:45.401 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2023-12-27T21:41:45.401 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 27/32 2023-12-27T21:41:45.401 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2023-12-27T21:41:45.401 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-12-27T21:41:45.401 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2023-12-27T21:41:45.402 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2023-12-27T21:41:57.975 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 32/32 2023-12-27T21:41:57.975 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-27T21:41:58.507 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-27T21:41:58.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-27T21:41:58.509 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-27T21:41:58.509 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-27T21:41:58.509 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2023-12-27T21:41:58.509 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-27T21:41:58.509 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-27T21:41:58.509 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-27T21:41:58.510 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-27T21:41:58.511 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:41:58.511 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:41:58.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2023-12-27T21:41:59.028 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:00.985 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2023-12-27T21:42:00.985 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:42:01.090 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-27T21:42:01.521 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-2.6-1.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-27T21:42:01.522 INFO:teuthology.orchestra.run.smithi196.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:01.523 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:42:01.676 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:42:01.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2023-12-27T21:42:02.072 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:03.771 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:42:03.939 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:42:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:42:03.940 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:42:04.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2023-12-27T21:42:04.418 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:04.567 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2023-12-27T21:42:04.879 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2023-12-27T21:42:06.616 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2023-12-27T21:42:07.216 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:42:07.239 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2023-12-27T21:42:07.388 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:42:07.389 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:42:07.389 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:42:07.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2023-12-27T21:42:07.872 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:09.453 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 85 k 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 85 k 2023-12-27T21:42:09.454 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 352 k 2023-12-27T21:42:09.455 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:42:09.720 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-16.2.14-417.gc5564c79.el8.noarch.rpm 322 kB/s | 85 kB 00:00 2023-12-27T21:42:09.720 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:42:09.720 INFO:teuthology.orchestra.run.smithi086.stdout:Total 320 kB/s | 85 kB 00:00 2023-12-27T21:42:09.721 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:42:09.728 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:42:09.728 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:42:09.730 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:42:09.739 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:42:09.800 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:42:10.078 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2023-12-27T21:42:10.467 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2023-12-27T21:42:11.523 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:11.739 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:11.942 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:12.974 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 85 k 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2023-12-27T21:42:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 352 k 2023-12-27T21:42:12.976 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:42:13.141 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-16.2.14-417.gc5564c79.el8.noarch.rpm 515 kB/s | 85 kB 00:00 2023-12-27T21:42:13.142 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:42:13.142 INFO:teuthology.orchestra.run.smithi196.stdout:Total 510 kB/s | 85 kB 00:00 2023-12-27T21:42:13.142 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:42:13.150 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:42:13.150 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:42:13.151 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:42:13.160 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:42:13.204 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:42:15.763 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:15.924 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:16.129 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:26.919 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:26.919 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:42:27.471 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:27.471 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:42:27.471 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:42:27.471 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:27.471 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:42:27.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2023-12-27T21:42:27.990 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:30.502 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2023-12-27T21:42:31.075 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2023-12-27T21:42:31.471 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/1 2023-12-27T21:42:31.471 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:42:31.999 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:31.999 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:42:31.999 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:42:31.999 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:32.000 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:42:32.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2023-12-27T21:42:32.515 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:33.299 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:42:33.300 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:42:33.300 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-417.gc5564c79.el8 ceph 161 k 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 161 k 2023-12-27T21:42:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 470 k 2023-12-27T21:42:33.302 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:42:33.543 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-16.2.14-417.gc5564c 667 kB/s | 161 kB 00:00 2023-12-27T21:42:33.543 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:42:33.543 INFO:teuthology.orchestra.run.smithi086.stdout:Total 664 kB/s | 161 kB 00:00 2023-12-27T21:42:33.544 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:42:33.555 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:42:33.555 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:42:33.608 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:42:33.617 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:42:33.734 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:42:33.906 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-417.gc5564c79. 1/1 2023-12-27T21:42:34.304 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-417.gc5564c79. 1/1 2023-12-27T21:42:34.776 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 18 kB/s | 4.1 kB 00:00 2023-12-27T21:42:35.195 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 23 kB/s | 4.5 kB 00:00 2023-12-27T21:42:37.800 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-417.gc5564c79.el8 ceph 161 k 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:42:37.801 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2023-12-27T21:42:37.802 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:37.802 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 161 k 2023-12-27T21:42:37.802 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 470 k 2023-12-27T21:42:37.802 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:42:37.997 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-16.2.14-417.gc5564c 829 kB/s | 161 kB 00:00 2023-12-27T21:42:37.997 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:42:37.998 INFO:teuthology.orchestra.run.smithi196.stdout:Total 823 kB/s | 161 kB 00:00 2023-12-27T21:42:37.998 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:42:38.009 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:42:38.009 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:42:38.062 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:42:38.071 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:42:38.185 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:42:38.356 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-417.gc5564c79. 1/1 2023-12-27T21:42:38.750 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-417.gc5564c79. 1/1 2023-12-27T21:42:52.206 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-417.gc5564c79. 1/1 2023-12-27T21:42:52.206 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:42:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:52.745 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:42:52.745 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:42:52.746 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:42:52.746 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:42:52.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2023-12-27T21:42:53.288 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:55.660 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2023-12-27T21:42:56.276 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2023-12-27T21:42:57.498 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-417.gc5564c79. 1/1 2023-12-27T21:42:57.498 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:42:58.030 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:58.031 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:42:58.031 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:42:58.031 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:42:58.031 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:42:58.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2023-12-27T21:42:58.363 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:42:58.537 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:42:58.538 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:42:58.538 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:42:58.546 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:42:58.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2023-12-27T21:42:59.032 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:43:00.521 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2023-12-27T21:43:00.815 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 kB/s | 4.5 kB 00:00 2023-12-27T21:43:01.298 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2023-12-27T21:43:01.882 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:43:03.114 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:43:03.291 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:43:03.292 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:43:03.293 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:43:03.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2023-12-27T21:43:03.767 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:43:04.217 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:=================================================================================================== 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:=================================================================================================== 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 3.1 M 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 24 k 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 15 k 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-27T21:43:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:=================================================================================================== 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.4 M 2023-12-27T21:43:04.219 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 22 M 2023-12-27T21:43:04.220 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:43:04.655 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): ceph-grafana-dashboards-16.2.14-417.gc55 60 kB/s | 24 kB 00:00 2023-12-27T21:43:04.712 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-prometheus-alerts-16.2.14-417.gc556 32 kB/s | 15 kB 00:00 2023-12-27T21:43:05.047 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 86 kB/s | 34 kB 00:00 2023-12-27T21:43:05.180 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 319 kB/s | 43 kB 00:00 2023-12-27T21:43:05.205 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 398 kB/s | 196 kB 00:00 2023-12-27T21:43:05.506 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-mgr-dashboard-16.2.14-417.gc5564c79 2.5 MB/s | 3.1 MB 00:01 2023-12-27T21:43:05.509 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:43:05.510 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.7 MB/s | 3.4 MB 00:01 2023-12-27T21:43:05.523 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:43:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:43:05.534 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:43:05.619 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:43:05.628 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:43:05.948 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:43:06.045 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2023-12-27T21:43:06.183 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-12-27T21:43:06.374 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-12-27T21:43:06.403 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2023-12-27T21:43:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-12-27T21:43:06.675 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-417.gc5564c79.el8.n 4/6 2023-12-27T21:43:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-417.gc5564c79.el8. 5/6 2023-12-27T21:43:07.994 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarc 6/6 2023-12-27T21:43:08.269 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarc 6/6 2023-12-27T21:43:08.269 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-417.gc5564c79.el8. 1/6 2023-12-27T21:43:08.269 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarc 2/6 2023-12-27T21:43:08.269 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-417.gc5564c79.el8.n 3/6 2023-12-27T21:43:08.269 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-12-27T21:43:08.269 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-12-27T21:43:08.902 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:43:08.902 INFO:teuthology.orchestra.run.smithi196.stdout:=================================================================================================== 2023-12-27T21:43:08.902 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:43:08.902 INFO:teuthology.orchestra.run.smithi196.stdout:=================================================================================================== 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 3.1 M 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 24 k 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 15 k 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:43:08.903 INFO:teuthology.orchestra.run.smithi196.stdout:=================================================================================================== 2023-12-27T21:43:08.904 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2023-12-27T21:43:08.904 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:43:08.904 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.4 M 2023-12-27T21:43:08.904 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 22 M 2023-12-27T21:43:08.904 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:43:09.840 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): ceph-grafana-dashboards-16.2.14-417.gc55 40 kB/s | 24 kB 00:00 2023-12-27T21:43:09.865 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): ceph-prometheus-alerts-16.2.14-417.gc556 24 kB/s | 15 kB 00:00 2023-12-27T21:43:09.915 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 447 kB/s | 34 kB 00:00 2023-12-27T21:43:09.940 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.6 MB/s | 196 kB 00:00 2023-12-27T21:43:10.174 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 165 kB/s | 43 kB 00:00 2023-12-27T21:43:11.366 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-mgr-dashboard-16.2.14-417.gc5564c79 1.5 MB/s | 3.1 MB 00:02 2023-12-27T21:43:11.369 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:43:11.369 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.4 MB/s | 3.4 MB 00:02 2023-12-27T21:43:11.383 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:43:11.394 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:43:11.394 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:43:11.477 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:43:11.485 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:43:11.757 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:43:11.943 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-12-27T21:43:12.221 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-12-27T21:43:12.393 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-12-27T21:43:12.514 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-417.gc5564c79.el8.n 4/6 2023-12-27T21:43:13.574 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-417.gc5564c79.el8. 5/6 2023-12-27T21:43:13.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarc 6/6 2023-12-27T21:43:14.086 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarc 6/6 2023-12-27T21:43:14.086 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-417.gc5564c79.el8. 1/6 2023-12-27T21:43:14.087 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarc 2/6 2023-12-27T21:43:14.087 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-417.gc5564c79.el8.n 3/6 2023-12-27T21:43:14.087 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-12-27T21:43:14.087 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-12-27T21:43:24.877 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-12-27T21:43:24.877 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:25.425 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:43:25.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-27T21:43:25.974 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:43:28.865 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 20 kB/s | 4.5 kB 00:00 2023-12-27T21:43:29.487 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2023-12-27T21:43:31.813 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:43:31.814 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================================= 2023-12-27T21:43:31.814 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:43:31.814 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================================= 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 8.6 M 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================================= 2023-12-27T21:43:31.815 INFO:teuthology.orchestra.run.smithi086.stdout:Install 8 Packages 2023-12-27T21:43:31.816 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:31.816 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2023-12-27T21:43:31.816 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 207 M 2023-12-27T21:43:31.816 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:43:31.968 INFO:teuthology.orchestra.run.smithi086.stdout:(1/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.5 MB/s | 225 kB 00:00 2023-12-27T21:43:32.077 INFO:teuthology.orchestra.run.smithi086.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2023-12-27T21:43:32.269 INFO:teuthology.orchestra.run.smithi086.stdout:(3/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 17 MB/s | 5.2 MB 00:00 2023-12-27T21:43:32.305 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-12-27T21:43:32.305 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:43:32.377 INFO:teuthology.orchestra.run.smithi086.stdout:(4/8): openblas-0.3.15-6.el8.x86_64.rpm 17 MB/s | 5.0 MB 00:00 2023-12-27T21:43:32.567 INFO:teuthology.orchestra.run.smithi086.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.3 MB/s | 645 kB 00:00 2023-12-27T21:43:32.703 INFO:teuthology.orchestra.run.smithi086.stdout:(6/8): python3-scipy-1.0.0-21.module+el8.9.0+20 32 MB/s | 14 MB 00:00 2023-12-27T21:43:32.728 INFO:teuthology.orchestra.run.smithi086.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2023-12-27T21:43:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:43:32.845 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:43:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:43:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:43:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:43:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-27T21:43:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-27T21:43:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-27T21:43:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:43:32.846 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:43:33.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-27T21:43:33.312 INFO:teuthology.orchestra.run.smithi086.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-41 5.7 MB/s | 8.6 MB 00:01 2023-12-27T21:43:33.313 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:43:33.313 INFO:teuthology.orchestra.run.smithi086.stdout:Total 25 MB/s | 37 MB 00:01 2023-12-27T21:43:33.374 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:43:33.563 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:43:33.580 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:43:33.580 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:43:33.892 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:43:33.901 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:43:34.451 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:43:34.581 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-12-27T21:43:34.722 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-12-27T21:43:34.915 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-12-27T21:43:35.481 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:43:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-12-27T21:43:35.723 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2023-12-27T21:43:35.909 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 21 kB/s | 4.5 kB 00:00 2023-12-27T21:43:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2023-12-27T21:43:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2023-12-27T21:43:37.254 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2023-12-27T21:43:37.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-27T21:43:37.789 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-27T21:43:38.406 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:43:38.407 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================================= 2023-12-27T21:43:38.407 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:43:38.407 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================================= 2023-12-27T21:43:38.407 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:43:38.407 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 8.6 M 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================================= 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout:Install 8 Packages 2023-12-27T21:43:38.408 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:43:38.409 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2023-12-27T21:43:38.409 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 207 M 2023-12-27T21:43:38.409 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:43:38.585 INFO:teuthology.orchestra.run.smithi196.stdout:(1/8): libquadmath-8.5.0-20.el8.x86_64.rpm 983 kB/s | 172 kB 00:00 2023-12-27T21:43:38.619 INFO:teuthology.orchestra.run.smithi196.stdout:(2/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.0 MB/s | 645 kB 00:00 2023-12-27T21:43:38.794 INFO:teuthology.orchestra.run.smithi196.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 18 MB/s | 3.7 MB 00:00 2023-12-27T21:43:38.820 INFO:teuthology.orchestra.run.smithi196.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2023-12-27T21:43:39.045 INFO:teuthology.orchestra.run.smithi196.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 21 MB/s | 5.2 MB 00:00 2023-12-27T21:43:39.162 INFO:teuthology.orchestra.run.smithi196.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 15 MB/s | 5.0 MB 00:00 2023-12-27T21:43:39.412 INFO:teuthology.orchestra.run.smithi196.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 37 MB/s | 14 MB 00:00 2023-12-27T21:43:39.529 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-27T21:43:39.879 INFO:teuthology.orchestra.run.smithi196.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-41 5.8 MB/s | 8.6 MB 00:01 2023-12-27T21:43:39.880 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:43:39.880 INFO:teuthology.orchestra.run.smithi196.stdout:Total 25 MB/s | 37 MB 00:01 2023-12-27T21:43:40.127 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:43:40.146 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:43:40.146 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:43:40.461 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:43:40.470 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:43:40.896 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2023-12-27T21:43:41.025 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:43:41.081 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c7 8/8 2023-12-27T21:43:41.168 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-12-27T21:43:41.286 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c7 8/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c7 1/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 2/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 3/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 4/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 5/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 6/8 2023-12-27T21:43:41.380 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-12-27T21:43:41.468 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-12-27T21:43:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-12-27T21:43:42.226 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2023-12-27T21:43:42.835 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2023-12-27T21:43:43.026 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2023-12-27T21:43:43.707 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2023-12-27T21:43:44.005 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-27T21:43:44.193 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-27T21:43:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-27T21:43:47.302 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2023-12-27T21:43:47.484 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c7 8/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c7 8/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c7 1/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2023-12-27T21:43:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2023-12-27T21:43:57.708 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-12-27T21:43:57.708 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:43:58.234 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:58.234 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:43:58.234 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:43:58.234 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-12-27T21:43:58.234 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-12-27T21:43:58.234 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.15-6.el8.x86_64 2023-12-27T21:43:58.234 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.15-6.el8.x86_64 2023-12-27T21:43:58.235 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-27T21:43:58.235 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-27T21:43:58.235 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2023-12-27T21:43:58.235 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:43:58.235 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:43:58.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2023-12-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:44:01.014 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 22 kB/s | 4.5 kB 00:00 2023-12-27T21:44:01.609 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2023-12-27T21:44:03.825 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================== 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================== 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 27 k 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-27T21:44:03.827 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================================== 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2023-12-27T21:44:03.828 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.9 M 2023-12-27T21:44:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 24 M 2023-12-27T21:44:03.829 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:44:04.071 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 77 kB/s | 16 kB 00:00 2023-12-27T21:44:04.096 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 139 kB/s | 33 kB 00:00 2023-12-27T21:44:04.180 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 749 kB/s | 81 kB 00:00 2023-12-27T21:44:04.255 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 858 kB/s | 64 kB 00:00 2023-12-27T21:44:04.322 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): ceph-mgr-rook-16.2.14-417.gc5564c79.el8 59 kB/s | 27 kB 00:00 2023-12-27T21:44:04.355 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 610 kB/s | 61 kB 00:00 2023-12-27T21:44:04.384 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2023-12-27T21:44:04.384 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:44:04.522 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-jsonpointer-1.10-11.el8.noarch. 100 kB/s | 20 kB 00:00 2023-12-27T21:44:04.548 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-pyasn1-modules-0.3.7-6.el8.noar 590 kB/s | 110 kB 00:00 2023-12-27T21:44:04.640 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 227 kB/s | 27 kB 00:00 2023-12-27T21:44:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 561 kB/s | 126 kB 00:00 2023-12-27T21:44:04.841 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 770 kB/s | 155 kB 00:00 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.15-6.el8.x86_64 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-threads-0.3.15-6.el8.x86_64 2023-12-27T21:44:04.921 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-27T21:44:04.922 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-27T21:44:04.922 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2023-12-27T21:44:04.922 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:04.922 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:44:04.932 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-requests-oauthlib-1.0.0-1.el8. 269 kB/s | 43 kB 00:00 2023-12-27T21:44:05.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2023-12-27T21:44:05.191 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.1 MB/s | 1.2 MB 00:01 2023-12-27T21:44:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:44:05.193 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-12-27T21:44:05.263 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:44:05.279 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:44:05.279 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:44:05.360 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:44:05.369 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:44:05.454 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:44:05.881 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:44:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-27T21:44:06.330 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-12-27T21:44:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-12-27T21:44:06.806 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-12-27T21:44:07.009 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-12-27T21:44:07.209 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-12-27T21:44:07.363 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-12-27T21:44:07.689 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-12-27T21:44:08.326 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2023-12-27T21:44:08.744 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 22 kB/s | 4.5 kB 00:00 2023-12-27T21:44:09.046 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-27T21:44:09.292 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-12-27T21:44:10.382 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-12-27T21:44:10.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-12-27T21:44:10.727 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 13/13 2023-12-27T21:44:11.006 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 13/13 2023-12-27T21:44:11.006 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 1/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 8/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-12-27T21:44:11.007 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-12-27T21:44:11.240 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================== 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================== 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 27 k 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-27T21:44:11.242 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================================== 2023-12-27T21:44:11.243 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2023-12-27T21:44:11.244 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:11.244 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 1.9 M 2023-12-27T21:44:11.244 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 24 M 2023-12-27T21:44:11.244 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:44:11.494 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 83 kB/s | 16 kB 00:00 2023-12-27T21:44:11.519 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 149 kB/s | 33 kB 00:00 2023-12-27T21:44:11.595 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): ceph-mgr-rook-16.2.14-417.gc5564c79.el8 92 kB/s | 27 kB 00:00 2023-12-27T21:44:11.628 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 607 kB/s | 81 kB 00:00 2023-12-27T21:44:11.712 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 733 kB/s | 61 kB 00:00 2023-12-27T21:44:11.854 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-kubernetes-11.0.0-6.el8.noarch. 3.5 MB/s | 1.2 MB 00:00 2023-12-27T21:44:11.871 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): python3-rsa-4.9-2.el8.noarch.rpm 233 kB/s | 64 kB 00:00 2023-12-27T21:44:11.896 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 844 kB/s | 155 kB 00:00 2023-12-27T21:44:12.096 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-pyasn1-modules-0.3.7-6.el8.noar 582 kB/s | 110 kB 00:00 2023-12-27T21:44:12.138 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 150 kB/s | 43 kB 00:00 2023-12-27T21:44:12.163 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 68 kB/s | 20 kB 00:00 2023-12-27T21:44:12.289 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 842 kB/s | 126 kB 00:00 2023-12-27T21:44:12.314 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-jsonpatch-1.21-2.el8.noarch.rp 122 kB/s | 27 kB 00:00 2023-12-27T21:44:12.315 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:44:12.315 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.8 MB/s | 1.9 MB 00:01 2023-12-27T21:44:12.379 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:44:12.394 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:44:12.394 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:44:12.474 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:44:12.482 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:44:12.976 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:44:13.204 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-27T21:44:13.398 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-12-27T21:44:13.561 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-12-27T21:44:13.744 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-12-27T21:44:13.948 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-12-27T21:44:15.486 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-27T21:44:15.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-27T21:44:16.028 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-12-27T21:44:16.235 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-27T21:44:16.460 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-12-27T21:44:17.512 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-12-27T21:44:17.728 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-12-27T21:44:17.853 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 13/13 2023-12-27T21:44:18.165 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 13/13 2023-12-27T21:44:18.165 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 1/13 2023-12-27T21:44:18.165 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-12-27T21:44:18.165 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-12-27T21:44:18.165 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 10/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 11/13 2023-12-27T21:44:18.166 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-27T21:44:27.337 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-12-27T21:44:27.337 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-27T21:44:27.911 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:27.912 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:44:28.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2023-12-27T21:44:28.416 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:44:31.443 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 20 kB/s | 4.5 kB 00:00 2023-12-27T21:44:32.035 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2023-12-27T21:44:34.376 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================================== 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================================== 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 116 k 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-27T21:44:34.377 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:34.378 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:44:34.378 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================================== 2023-12-27T21:44:34.378 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2023-12-27T21:44:34.378 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:34.378 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 5.8 M 2023-12-27T21:44:34.378 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 24 M 2023-12-27T21:44:34.378 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:44:34.635 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-apipkg-1.5-6.el8.noarch.rpm 177 kB/s | 21 kB 00:00 2023-12-27T21:44:34.694 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-mgr-cephadm-16.2.14-417.gc5564c79.e 652 kB/s | 116 kB 00:00 2023-12-27T21:44:34.719 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-remoto-1.2.1-1.el8.noarch.rpm 584 kB/s | 49 kB 00:00 2023-12-27T21:44:34.744 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.5 MB/s | 355 kB 00:00 2023-12-27T21:44:34.894 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.0 MB/s | 538 kB 00:00 2023-12-27T21:44:35.020 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2023-12-27T21:44:35.021 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:44:35.021 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.1 MB/s | 5.8 MB 00:00 2023-12-27T21:44:35.075 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:44:35.085 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:44:35.085 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:44:35.174 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:44:35.182 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:44:36.175 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:44:36.536 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/6 2023-12-27T21:44:36.788 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/6 2023-12-27T21:44:37.064 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2023-12-27T21:44:37.385 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2023-12-27T21:44:37.598 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 5/6 2023-12-27T21:44:37.807 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 6/6 2023-12-27T21:44:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 6/6 2023-12-27T21:44:38.083 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/6 2023-12-27T21:44:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2023-12-27T21:44:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2023-12-27T21:44:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 4/6 2023-12-27T21:44:38.084 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/6 2023-12-27T21:44:39.406 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 13/13 2023-12-27T21:44:39.407 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-27T21:44:39.966 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-27T21:44:39.967 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-27T21:44:39.967 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-27T21:44:39.967 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-27T21:44:39.967 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-27T21:44:39.967 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:39.967 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:44:40.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2023-12-27T21:44:40.479 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:44:43.099 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 kB/s | 4.1 kB 00:00 2023-12-27T21:44:43.462 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2023-12-27T21:44:45.955 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:44:45.956 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================================== 2023-12-27T21:44:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:44:45.957 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================================== 2023-12-27T21:44:45.957 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:44:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:16.2.14-417.gc5564c79.el8 ceph-noarch 116 k 2023-12-27T21:44:45.957 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:44:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-27T21:44:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-12-27T21:44:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-27T21:44:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-12-27T21:44:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-27T21:44:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:45.958 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:44:45.958 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================================== 2023-12-27T21:44:45.959 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2023-12-27T21:44:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:44:45.960 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 5.8 M 2023-12-27T21:44:45.960 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 24 M 2023-12-27T21:44:45.960 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:44:46.323 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): ceph-mgr-cephadm-16.2.14-417.gc5564c79.e 398 kB/s | 116 kB 00:00 2023-12-27T21:44:46.908 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 24 kB/s | 21 kB 00:00 2023-12-27T21:44:47.176 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-12-27T21:44:47.350 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.0 MB/s | 538 kB 00:00 2023-12-27T21:44:47.375 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 46 kB/s | 49 kB 00:01 2023-12-27T21:44:47.717 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 211 kB/s | 355 kB 00:01 2023-12-27T21:44:47.720 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:44:47.720 INFO:teuthology.orchestra.run.smithi196.stdout:Total 3.3 MB/s | 5.8 MB 00:01 2023-12-27T21:44:47.778 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:44:47.789 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:44:47.789 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:44:47.878 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:44:47.887 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:44:48.869 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:44:49.218 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/6 2023-12-27T21:44:49.437 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/6 2023-12-27T21:44:49.748 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 3/6 2023-12-27T21:44:50.025 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 4/6 2023-12-27T21:44:50.297 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 5/6 2023-12-27T21:44:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 6/6 2023-12-27T21:44:50.757 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 6/6 2023-12-27T21:44:50.757 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 1/6 2023-12-27T21:44:50.757 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 2/6 2023-12-27T21:44:50.757 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 3/6 2023-12-27T21:44:50.757 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 4/6 2023-12-27T21:44:50.757 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/6 2023-12-27T21:44:52.993 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 6/6 2023-12-27T21:44:52.994 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:53.564 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:44:53.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2023-12-27T21:44:54.100 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:44:56.842 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 17 kB/s | 4.5 kB 00:00 2023-12-27T21:44:57.420 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 39 kB/s | 4.1 kB 00:00 2023-12-27T21:44:59.762 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:16.2.14-417.gc5564c79.el8 ceph 911 k 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:59.763 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:44:59.764 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:44:59.764 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2023-12-27T21:44:59.764 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:44:59.764 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 994 k 2023-12-27T21:44:59.764 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.8 M 2023-12-27T21:44:59.764 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:44:59.898 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 623 kB/s | 83 kB 00:00 2023-12-27T21:45:00.173 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-16.2.14-417.gc5564c79.el8.x86_ 2.2 MB/s | 911 kB 00:00 2023-12-27T21:45:00.174 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:45:00.174 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.4 MB/s | 994 kB 00:00 2023-12-27T21:45:00.186 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:45:00.206 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:45:00.206 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:45:00.292 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:45:00.300 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:45:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:45:00.703 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-27T21:45:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 2/2 2023-12-27T21:45:01.259 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 2/2 2023-12-27T21:45:01.259 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/2 2023-12-27T21:45:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 6/6 2023-12-27T21:45:10.430 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:16.2.14-417.gc5564c79.el8.noarch 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:10.964 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:45:11.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2023-12-27T21:45:11.495 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:45:14.079 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2023-12-27T21:45:14.419 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2023-12-27T21:45:16.908 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:45:16.909 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:45:16.909 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-12-27T21:45:16.909 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:45:16.909 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:45:16.909 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 2:16.2.14-417.gc5564c79.el8 ceph 911 k 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2023-12-27T21:45:16.910 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:16.911 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 994 k 2023-12-27T21:45:16.911 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.8 M 2023-12-27T21:45:16.911 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 503 kB/s | 83 kB 00:00 2023-12-27T21:45:17.310 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-16.2.14-417.gc5564c79.el8.x86_ 2.2 MB/s | 911 kB 00:00 2023-12-27T21:45:17.310 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:45:17.310 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.4 MB/s | 994 kB 00:00 2023-12-27T21:45:17.322 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:45:17.342 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:45:17.343 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:45:17.427 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:45:17.435 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:45:17.598 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:45:17.850 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-27T21:45:18.025 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 2/2 2023-12-27T21:45:18.406 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 2/2 2023-12-27T21:45:18.406 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/2 2023-12-27T21:45:19.651 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-27T21:45:19.652 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:45:20.189 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:20.189 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:45:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-27T21:45:20.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:20.190 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:45:20.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2023-12-27T21:45:20.698 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:45:23.368 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2023-12-27T21:45:23.971 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2023-12-27T21:45:26.305 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:45:26.306 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:45:26.306 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-27T21:45:26.306 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:16.2.14-417.gc5564c79.el8 ceph 131 k 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:26.307 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 131 k 2023-12-27T21:45:26.308 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 432 k 2023-12-27T21:45:26.308 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:45:26.620 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-16.2.14-417.gc5564c79.el8.x86_64 418 kB/s | 131 kB 00:00 2023-12-27T21:45:26.621 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:45:26.621 INFO:teuthology.orchestra.run.smithi086.stdout:Total 416 kB/s | 131 kB 00:00 2023-12-27T21:45:26.621 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:45:26.627 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:45:26.627 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:45:26.683 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:45:26.692 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:45:26.804 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:45:26.999 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:45:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:45:34.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-27T21:45:34.461 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:45:35.006 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:35.007 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:45:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-27T21:45:35.007 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:35.007 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:45:35.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2023-12-27T21:45:35.525 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:45:37.647 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 kB/s | 4.1 kB 00:00 2023-12-27T21:45:38.157 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 15 kB/s | 4.5 kB 00:00 2023-12-27T21:45:40.758 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:45:40.758 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:16.2.14-417.gc5564c79.el8 ceph 131 k 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2023-12-27T21:45:40.759 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:45:40.760 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 131 k 2023-12-27T21:45:40.760 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 432 k 2023-12-27T21:45:40.760 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:45:40.939 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-16.2.14-417.gc5564c79.el8.x86_64 734 kB/s | 131 kB 00:00 2023-12-27T21:45:40.939 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:45:40.939 INFO:teuthology.orchestra.run.smithi196.stdout:Total 729 kB/s | 131 kB 00:00 2023-12-27T21:45:40.940 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:45:40.946 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:45:40.946 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:45:41.003 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:45:41.012 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:45:41.130 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:45:41.311 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:45:41.574 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:45:43.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:45:43.967 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:45:44.501 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:44.502 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:45:44.502 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:45:44.502 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:44.502 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:45:44.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2023-12-27T21:45:45.006 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:45:47.410 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2023-12-27T21:45:48.007 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2023-12-27T21:45:50.166 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:45:50.341 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:45:50.342 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:45:50.342 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:45:50.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2023-12-27T21:45:50.841 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:45:52.919 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2023-12-27T21:45:53.557 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2023-12-27T21:45:55.797 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:45:55.797 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:45:55.800 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-27T21:45:55.800 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:45:55.800 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:45:55.800 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:16.2.14-417.gc5564c79.el8 ceph 25 k 2023-12-27T21:45:55.800 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:55.801 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:45:55.801 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:45:55.801 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-27T21:45:55.801 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:45:55.801 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 25 k 2023-12-27T21:45:55.801 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 107 k 2023-12-27T21:45:55.802 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:45:56.032 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-16.2.14-417.gc5564c79.el8.x86_6 109 kB/s | 25 kB 00:00 2023-12-27T21:45:56.033 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:45:56.033 INFO:teuthology.orchestra.run.smithi086.stdout:Total 109 kB/s | 25 kB 00:00 2023-12-27T21:45:56.033 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:45:56.037 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:45:56.037 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:45:56.040 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:45:56.048 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:45:56.096 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:45:56.229 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:45:56.477 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:01.614 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:01.614 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:46:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:46:02.177 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:46:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:46:02.177 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:46:02.177 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:46:02.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2023-12-27T21:46:02.706 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:05.108 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2023-12-27T21:46:05.442 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2023-12-27T21:46:07.780 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:07.956 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:46:07.958 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:46:07.958 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:46:08.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2023-12-27T21:46:08.455 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:10.815 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:10.815 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:46:10.873 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2023-12-27T21:46:11.218 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2023-12-27T21:46:11.358 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:46:11.358 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:46:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:46:11.359 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:46:11.359 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:46:11.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2023-12-27T21:46:11.880 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:13.716 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:46:13.717 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:46:13.717 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2023-12-27T21:46:13.717 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:46:13.717 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:46:13.717 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:16.2.14-417.gc5564c79.el8 ceph 25 k 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 25 k 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 107 k 2023-12-27T21:46:13.718 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:46:13.867 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-16.2.14-417.gc5564c79.el8.x86_6 172 kB/s | 25 kB 00:00 2023-12-27T21:46:13.867 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:46:13.867 INFO:teuthology.orchestra.run.smithi196.stdout:Total 170 kB/s | 25 kB 00:00 2023-12-27T21:46:13.868 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:46:13.871 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:46:13.871 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:46:13.873 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:46:13.882 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:46:13.932 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:46:14.003 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2023-12-27T21:46:14.080 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:14.331 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:14.589 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2023-12-27T21:46:16.649 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:16.823 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:46:16.825 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:46:16.825 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:46:16.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2023-12-27T21:46:17.305 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:19.630 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2023-12-27T21:46:20.340 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 16 kB/s | 4.1 kB 00:00 2023-12-27T21:46:22.393 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:22.571 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:46:22.572 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:46:22.572 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:46:22.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2023-12-27T21:46:23.068 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:25.409 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2023-12-27T21:46:26.033 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2023-12-27T21:46:28.107 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:28.286 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:46:28.287 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:46:28.287 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:46:28.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2023-12-27T21:46:28.777 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:31.322 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2023-12-27T21:46:32.039 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 16 kB/s | 4.1 kB 00:00 2023-12-27T21:46:32.091 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:32.092 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:46:32.642 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:46:32.642 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:46:32.642 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:46:32.642 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:46:32.642 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:46:32.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2023-12-27T21:46:33.184 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:34.096 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:34.271 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:46:34.272 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:46:34.272 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:46:34.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2023-12-27T21:46:34.759 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:35.446 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:46:35.792 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2023-12-27T21:46:36.966 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2023-12-27T21:46:37.529 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 4.1 kB 00:00 2023-12-27T21:46:38.124 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:38.301 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:46:38.302 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:46:38.303 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:46:38.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2023-12-27T21:46:38.807 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:39.582 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:39.755 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:46:39.757 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:46:39.757 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:46:39.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2023-12-27T21:46:40.247 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:40.910 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2023-12-27T21:46:41.296 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2023-12-27T21:46:42.443 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2023-12-27T21:46:42.986 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 kB/s | 4.1 kB 00:00 2023-12-27T21:46:43.710 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:43.889 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:46:43.890 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:46:43.890 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:46:44.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2023-12-27T21:46:44.381 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:45.041 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:45.214 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:46:45.215 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:46:45.215 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:46:45.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2023-12-27T21:46:45.699 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:46.789 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:46:47.108 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2023-12-27T21:46:47.890 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2023-12-27T21:46:48.539 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2023-12-27T21:46:49.426 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:49.603 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:46:49.604 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:46:49.604 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:46:49.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2023-12-27T21:46:50.089 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:50.873 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:16.2.14-417.gc5564c79.el8 ceph 88 k 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 88 k 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 240 k 2023-12-27T21:46:50.874 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:46:51.313 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-16.2.14-417.gc5564c79.el8.x86_64.rpm 202 kB/s | 88 kB 00:00 2023-12-27T21:46:51.313 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:46:51.313 INFO:teuthology.orchestra.run.smithi086.stdout:Total 201 kB/s | 88 kB 00:00 2023-12-27T21:46:51.313 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:46:51.323 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:46:51.323 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:46:51.381 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:46:51.389 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:46:51.531 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:46:51.696 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:51.961 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:46:52.058 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2023-12-27T21:46:52.412 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2023-12-27T21:46:54.836 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:46:55.012 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:46:55.013 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:46:55.013 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:46:55.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2023-12-27T21:46:55.511 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:46:57.716 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2023-12-27T21:46:58.269 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 13 kB/s | 4.5 kB 00:00 2023-12-27T21:47:00.637 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:47:00.817 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:47:00.818 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:47:00.818 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:47:00.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2023-12-27T21:47:01.322 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:47:03.307 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2023-12-27T21:47:03.640 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2023-12-27T21:47:05.947 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:47:06.121 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:47:06.122 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:47:06.122 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:47:06.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2023-12-27T21:47:06.617 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:47:06.774 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:06.774 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:47:07.322 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:07.322 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:47:07.322 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:47:07.322 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:07.322 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:47:07.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2023-12-27T21:47:07.859 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:47:08.704 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 20 kB/s | 4.1 kB 00:00 2023-12-27T21:47:09.019 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 kB/s | 4.5 kB 00:00 2023-12-27T21:47:09.887 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2023-12-27T21:47:10.571 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 18 kB/s | 4.1 kB 00:00 2023-12-27T21:47:11.513 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:47:11.513 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:16.2.14-417.gc5564c79.el8 ceph 88 k 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2023-12-27T21:47:11.514 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:11.515 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 88 k 2023-12-27T21:47:11.515 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 240 k 2023-12-27T21:47:11.515 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:47:11.676 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-16.2.14-417.gc5564c79.el8.x86_64.rpm 547 kB/s | 88 kB 00:00 2023-12-27T21:47:11.677 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:47:11.677 INFO:teuthology.orchestra.run.smithi196.stdout:Total 543 kB/s | 88 kB 00:00 2023-12-27T21:47:11.677 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:47:11.687 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:47:11.687 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:47:11.744 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:47:11.753 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:47:11.865 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:47:12.035 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:12.295 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:12.786 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:16.2.14-417.gc5564c79.el8 ceph 3.7 M 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:47:12.787 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:47:12.788 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-27T21:47:12.788 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:12.788 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.7 M 2023-12-27T21:47:12.788 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 15 M 2023-12-27T21:47:12.788 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:47:13.565 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-16.2.14-417.gc5564c79.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2023-12-27T21:47:13.565 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:47:13.566 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.7 MB/s | 3.7 MB 00:00 2023-12-27T21:47:13.566 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:47:13.578 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:47:13.578 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:47:13.648 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:47:13.657 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:47:14.099 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:47:14.270 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:14.684 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:25.337 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:25.337 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:47:25.886 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:25.886 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:47:25.886 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:47:25.886 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:25.886 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:47:26.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2023-12-27T21:47:26.408 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:47:28.643 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2023-12-27T21:47:28.993 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2023-12-27T21:47:30.339 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:30.339 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:47:30.871 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:30.871 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:47:30.871 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:47:30.871 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:30.871 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:47:31.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2023-12-27T21:47:31.400 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:47:31.490 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:16.2.14-417.gc5564c79.el8 ceph 3.7 M 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:31.491 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2023-12-27T21:47:31.492 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:31.492 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.7 M 2023-12-27T21:47:31.492 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 15 M 2023-12-27T21:47:31.492 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:47:32.419 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-16.2.14-417.gc5564c79.el8.x86_64.rpm 4.0 MB/s | 3.7 MB 00:00 2023-12-27T21:47:32.419 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:47:32.420 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.0 MB/s | 3.7 MB 00:00 2023-12-27T21:47:32.420 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:47:32.433 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:47:32.433 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:47:32.506 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:47:32.515 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:47:32.959 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:47:33.133 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:33.563 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:33.776 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2023-12-27T21:47:34.360 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:47:36.705 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:47:36.706 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:47:36.706 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:16.2.14-417.gc5564c79.el8 ceph 175 k 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 175 k 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 523 k 2023-12-27T21:47:36.707 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-12-27T21:47:37.043 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-16.2.14-417.gc5564c79.el8.x86_64.rpm 522 kB/s | 175 kB 00:00 2023-12-27T21:47:37.044 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:47:37.044 INFO:teuthology.orchestra.run.smithi086.stdout:Total 520 kB/s | 175 kB 00:00 2023-12-27T21:47:37.044 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-12-27T21:47:37.054 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-12-27T21:47:37.054 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-12-27T21:47:37.111 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-12-27T21:47:37.119 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-12-27T21:47:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-12-27T21:47:37.423 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:37.680 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:47.757 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:47.757 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:47:48.296 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:48.296 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:47:48.296 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:47:48.296 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:48.296 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:47:48.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2023-12-27T21:47:48.825 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:47:51.093 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 21 kB/s | 4.1 kB 00:00 2023-12-27T21:47:51.464 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2023-12-27T21:47:54.069 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:16.2.14-417.gc5564c79.el8 ceph 175 k 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2023-12-27T21:47:54.070 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2023-12-27T21:47:54.071 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:47:54.071 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 175 k 2023-12-27T21:47:54.071 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 523 k 2023-12-27T21:47:54.071 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-12-27T21:47:54.073 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:54.073 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-12-27T21:47:54.254 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-16.2.14-417.gc5564c79.el8.x86_64.rpm 961 kB/s | 175 kB 00:00 2023-12-27T21:47:54.254 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-12-27T21:47:54.254 INFO:teuthology.orchestra.run.smithi196.stdout:Total 955 kB/s | 175 kB 00:00 2023-12-27T21:47:54.255 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-12-27T21:47:54.265 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-12-27T21:47:54.265 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-12-27T21:47:54.322 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-12-27T21:47:54.331 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-12-27T21:47:54.447 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-12-27T21:47:54.618 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:54.618 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-12-27T21:47:54.618 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:47:54.618 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:47:54.618 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:47:54.635 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:54.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2023-12-27T21:47:54.897 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:47:55.135 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-12-27T21:47:56.973 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2023-12-27T21:47:57.566 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2023-12-27T21:47:59.612 INFO:teuthology.orchestra.run.smithi086.stdout:Package rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:47:59.791 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-12-27T21:47:59.792 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-12-27T21:47:59.792 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-12-27T21:47:59.930 DEBUG:teuthology.parallel:result is None 2023-12-27T21:48:11.781 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 1/1 2023-12-27T21:48:11.782 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-12-27T21:48:12.315 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:12.315 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-12-27T21:48:12.315 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:16.2.14-417.gc5564c79.el8.x86_64 2023-12-27T21:48:12.316 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:12.316 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:48:12.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2023-12-27T21:48:12.857 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-12-27T21:48:14.935 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2023-12-27T21:48:15.321 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2023-12-27T21:48:17.638 INFO:teuthology.orchestra.run.smithi196.stdout:Package rbd-mirror-2:16.2.14-417.gc5564c79.el8.x86_64 is already installed. 2023-12-27T21:48:17.814 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-12-27T21:48:17.815 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-12-27T21:48:17.815 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-12-27T21:48:17.963 DEBUG:teuthology.parallel:result is None 2023-12-27T21:48:17.964 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c5564c7988cbaadc3382253af5843a8595347c2d 2023-12-27T21:48:18.112 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-27T21:48:18.138 INFO:teuthology.orchestra.run.smithi086.stdout:16.2.14-417.gc5564c79.el8 2023-12-27T21:48:18.139 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-417.gc5564c79.el8 2023-12-27T21:48:18.139 INFO:teuthology.task.install:The correct ceph version 16.2.14-417.gc5564c79 is installed. 2023-12-27T21:48:18.141 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c5564c7988cbaadc3382253af5843a8595347c2d 2023-12-27T21:48:18.386 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-27T21:48:18.409 INFO:teuthology.orchestra.run.smithi196.stdout:16.2.14-417.gc5564c79.el8 2023-12-27T21:48:18.409 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-417.gc5564c79.el8 2023-12-27T21:48:18.409 INFO:teuthology.task.install:The correct ceph version 16.2.14-417.gc5564c79 is installed. 2023-12-27T21:48:18.411 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-27T21:48:18.411 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:18.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-27T21:48:18.443 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:18.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-27T21:48:18.477 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-27T21:48:18.477 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:18.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2023-12-27T21:48:18.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-27T21:48:18.582 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:18.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2023-12-27T21:48:18.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-27T21:48:18.682 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-27T21:48:18.682 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:18.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-27T21:48:18.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-27T21:48:18.779 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:18.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-27T21:48:18.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-27T21:48:18.875 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-27T21:48:18.875 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:18.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2023-12-27T21:48:18.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-27T21:48:18.969 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:18.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2023-12-27T21:48:18.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-27T21:48:19.064 INFO:teuthology.run_tasks:Running task ceph... 2023-12-27T21:48:19.134 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-27T21:48:19.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2023-12-27T21:48:19.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2023-12-27T21:48:19.164 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-27T21:48:19.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-27T21:48:19.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-27T21:48:19.231 INFO:tasks.ceph:Creating extra log directories... 2023-12-27T21:48:19.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-27T21:48:19.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-27T21:48:19.318 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-27T21:48:19.318 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-27T21:48:19.318 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502301', 'branch': 'wip-yuri7-testing-2023-12-27-1008-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7502301', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2023-12-27-1008-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri7-testing-2023-12-27-1008-pacific', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5573, 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri7-testing-2023-12-27-1008-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d', 'targets': {'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOblRzH4CISFZgKXd44gHxbtkO20DUUrc3Tk8ZWISfJl0F8R7ZnFxuu/tw3Bx18hMtCqyI+gKAQQNinSK5R6+80=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQEeKbf/5+MFgp1CU514Rt/z8u7LW2e6ayiIfhvGzRKJnbTd60HsQS9USBrvGz6/7ODtWja1GHRU4zoLdGn7aU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '01dd171a561ca33631bf0a38b105879779f72410', 'timestamp': '2023-12-27_20:59:12', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-27T21:48:19.319 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-27T21:48:19.339 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-27T21:48:19.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2023-12-27T21:48:19.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2023-12-27T21:48:19.425 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:19.425 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2023-12-27T21:48:19.479 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-27T21:48:19.479 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2023-12-27T21:48:19.536 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-27T21:48:19.536 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:19.536 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 126247 Links: 1 2023-12-27T21:48:19.537 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:19.537 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:19.537 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-27 21:47:14.454630336 +0000 2023-12-27T21:48:19.537 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-27 21:38:25.556214177 +0000 2023-12-27T21:48:19.537 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-27 21:38:25.556214177 +0000 2023-12-27T21:48:19.537 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-12-27T21:48:19.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-27T21:48:19.602 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-27T21:48:19.603 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-27T21:48:19.603 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000292598 s, 1.7 MB/s 2023-12-27T21:48:19.604 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-27T21:48:19.662 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 129289 Links: 1 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-27 21:47:14.454630336 +0000 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-27 21:38:26.064203161 +0000 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-27 21:38:26.064203161 +0000 2023-12-27T21:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-12-27T21:48:19.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-27T21:48:19.787 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-27T21:48:19.787 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-27T21:48:19.787 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000223492 s, 2.3 MB/s 2023-12-27T21:48:19.787 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-27T21:48:19.844 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 125933 Links: 1 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-27 21:47:14.454630336 +0000 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-27 21:38:26.607191387 +0000 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-27 21:38:26.607191387 +0000 2023-12-27T21:48:19.902 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-12-27T21:48:19.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-27T21:48:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-27T21:48:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-27T21:48:19.968 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000223527 s, 2.3 MB/s 2023-12-27T21:48:19.969 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-27T21:48:20.029 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 130538 Links: 1 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-12-27 21:47:14.454630336 +0000 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-12-27 21:38:27.111180458 +0000 2023-12-27T21:48:20.087 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-12-27 21:38:27.111180458 +0000 2023-12-27T21:48:20.088 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-12-27T21:48:20.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-27T21:48:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-12-27T21:48:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-12-27T21:48:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000235596 s, 2.2 MB/s 2023-12-27T21:48:20.154 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-27T21:48:20.211 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:20.211 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-27T21:48:20.212 INFO:tasks.ceph:Generating config... 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client] log to stderr = False 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.213 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.214 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.214 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.214 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-27T21:48:20.214 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-27T21:48:20.214 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-27T21:48:20.214 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-27T21:48:20.214 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-27T21:48:20.214 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-27T21:48:20.214 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-27T21:48:20.214 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-27T21:48:20.214 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-27T21:48:20.214 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-27T21:48:20.214 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-27T21:48:20.214 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-27T21:48:20.214 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-27T21:48:20.215 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-27T21:48:20.215 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-27T21:48:20.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-27T21:48:20.295 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/cluster1.keyring 2023-12-27T21:48:20.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-27T21:48:20.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-27T21:48:20.445 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.86')] 2023-12-27T21:48:20.446 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.86', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-27T21:48:20.446 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:20.446 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-27T21:48:20.502 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.86 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:20.579 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: generated fsid b14702b5-3263-4304-8c6f-fcd3bab3acd5 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 0 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:fsid b14702b5-3263-4304-8c6f-fcd3bab3acd5 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:last_changed 2023-12-27T21:48:20.581097+0000 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:created 2023-12-27T21:48:20.581097+0000 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:min_mon_release 0 (unknown) 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:election_strategy: 1 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2023-12-27T21:48:20.580 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-27T21:48:20.581 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-27T21:48:20.638 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID b14702b5-3263-4304-8c6f-fcd3bab3acd5... 2023-12-27T21:48:20.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-27T21:48:20.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-27T21:48:20.724 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2023-12-27T21:48:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2023-12-27T21:48:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-27T21:48:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2023-12-27T21:48:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2023-12-27T21:48:20.724 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.726 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = 172.21.15.86 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 5000 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = b14702b5-3263-4304-8c6f-fcd3bab3acd5 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.727 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2023-12-27T21:48:20.728 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: bdev async discard = True 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: bdev enable discard = True 2023-12-27T21:48:20.729 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore allocator = stupid 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 1/20 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 1/20 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 4/10 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2023-12-27T21:48:20.730 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:20.731 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2023-12-27T21:48:20.732 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: debug rbd = 20 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: debug rbd_mirror = 15 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: log to stderr = False 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: rbd default features = 125 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: rbd mirror image policy type = none 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.0] 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.1] 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.733 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.2] 2023-12-27T21:48:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.734 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.3] 2023-12-27T21:48:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.734 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.735 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2023-12-27T21:48:20.736 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2023-12-27T21:48:20.737 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = 172.21.15.86 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: mon client directed command retry = 5 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: ms inject socket failures = 5000 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = b14702b5-3263-4304-8c6f-fcd3bab3acd5 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2023-12-27T21:48:20.738 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2023-12-27T21:48:20.739 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: bdev async discard = True 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: bdev enable discard = True 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore allocator = stupid 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 1/20 2023-12-27T21:48:20.740 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 1/20 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 4/10 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2023-12-27T21:48:20.741 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:20.742 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-27T21:48:20.743 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: debug rbd = 20 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: debug rbd_mirror = 15 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: log to stderr = False 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: rbd default features = 125 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: rbd mirror image policy type = none 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.0] 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.1] 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.2] 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.744 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.745 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.3] 2023-12-27T21:48:20.745 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:20.745 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:20.748 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-27T21:48:20.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-27T21:48:20.837 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-27T21:48:20.837 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:20.837 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-27T21:48:20.854 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:20.854 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-27T21:48:20.911 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2023-12-27T21:48:20.911 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:20.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-27T21:48:20.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-27T21:48:20.988 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:20.988 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:21.045 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2023-12-27T21:48:21.045 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:21.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-27T21:48:21.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-27T21:48:21.090 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:21.090 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:21.150 INFO:tasks.ceph:Setting up mon nodes... 2023-12-27T21:48:21.151 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-27T21:48:21.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-27T21:48:21.200 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-27T21:48:21.202 INFO:tasks.ceph:Setting up mds nodes... 2023-12-27T21:48:21.202 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-27T21:48:21.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-27T21:48:21.242 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-27T21:48:21.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-27T21:48:21.315 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-27T21:48:21.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-27T21:48:21.372 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-27T21:48:21.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-27T21:48:21.466 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-27T21:48:21.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-27T21:48:21.558 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-27T21:48:21.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-27T21:48:21.650 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-27T21:48:21.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-27T21:48:21.742 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-27T21:48:21.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-27T21:48:21.834 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-27T21:48:21.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-27T21:48:21.926 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-27T21:48:21.939 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-27T21:48:21.939 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-27T21:48:21.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-27T21:48:21.968 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:21.968 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-27T21:48:21.968 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-27T21:48:21.969 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-27T21:48:22.045 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-27T21:48:22.045 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-27T21:48:22.045 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-27T21:48:22.045 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-27T21:48:22.046 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-27T21:48:22.046 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-12-27T21:48:22.046 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-27T21:48:22.046 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-27T21:48:22.046 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-27T21:48:22.046 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-12-27T21:48:22.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-12-27T21:48:22.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-27T21:48:22.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-27T21:48:22.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-27T21:48:22.796 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:22.796 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-27T21:48:22.796 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-27T21:48:22.796 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-27T21:48:22.871 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-27T21:48:22.871 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-27T21:48:22.871 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-27T21:48:22.871 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-27T21:48:22.871 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-27T21:48:22.871 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-12-27T21:48:22.871 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-27T21:48:22.872 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-27T21:48:22.872 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-27T21:48:22.872 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-27T21:48:23.276 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-12-27T21:48:23.279 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-12-27T21:48:23.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-27T21:48:23.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-27T21:48:23.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-27T21:48:23.500 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:23.500 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-27T21:48:23.500 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2023-12-27T21:48:23.500 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-27T21:48:23.576 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-27T21:48:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-27T21:48:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-27T21:48:23.576 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-27T21:48:23.577 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-27T21:48:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-12-27T21:48:23.577 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-27T21:48:23.577 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-27T21:48:23.577 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-27T21:48:23.577 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-27T21:48:24.026 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-12-27T21:48:24.027 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-12-27T21:48:24.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-27T21:48:24.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-27T21:48:24.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:24.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:24.322+0000 7fb613289380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-27T21:48:24.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:24.323+0000 7fb613289380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-27T21:48:24.322 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:24.323+0000 7fb613289380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-27T21:48:24.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:24.323+0000 7fb613289380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-27T21:48:24.323 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:24.323+0000 7fb613289380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-27T21:48:24.328 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:24.329+0000 7fb613289380 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-27T21:48:27.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-27T21:48:27.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:27.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:27.285+0000 7f82cbca1380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-27T21:48:27.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:27.285+0000 7f82cbca1380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-27T21:48:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:27.285+0000 7f82cbca1380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-27T21:48:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:27.285+0000 7f82cbca1380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-27T21:48:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:27.285+0000 7f82cbca1380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-27T21:48:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:27.286+0000 7f82cbca1380 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-27T21:48:30.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-27T21:48:30.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:30.257+0000 7fd9f882f380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-27T21:48:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:30.258+0000 7fd9f882f380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-27T21:48:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:30.258+0000 7fd9f882f380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-27T21:48:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:30.258+0000 7fd9f882f380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-27T21:48:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:30.258+0000 7fd9f882f380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-27T21:48:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2023-12-27T21:48:30.258+0000 7fd9f882f380 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-27T21:48:33.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-27T21:48:33.122 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-27T21:48:33.122 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:33.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-27T21:48:33.189 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:33.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-27T21:48:33.256 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:33.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-27T21:48:33.323 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:33.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-27T21:48:33.390 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:33.390 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-27T21:48:33.447 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.448 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-27T21:48:33.468 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.468 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-27T21:48:33.529 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.530 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-27T21:48:33.589 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.589 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-27T21:48:33.649 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.649 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-27T21:48:33.708 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.708 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-27T21:48:33.767 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.767 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-27T21:48:33.826 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:33.826 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-27T21:48:33.885 INFO:tasks.ceph:Adding keys to all mons... 2023-12-27T21:48:33.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-27T21:48:33.912 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xlTe0IDBAAGWZMp6grfUfYAB3/DoHJEQ== 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCom4xlbFBJExAAWBIzeICNc6ZNCtnL4tShSQ== 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCrm4xlKHgIERAANuGpr6HpJ+jO7hamKL8KGg== 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCum4xl22NhDxAA5xqe5Stg4mvMzW7su9Jr3g== 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xlj3CDDhAAIRo+hwob2R/Sr60wJJIpNQ== 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror] 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xlaEjdEhAAW9oojV/NrSHEoCoqXu3z9g== 2023-12-27T21:48:33.913 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.0] 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xlNGpEFhAAGmpvpNI4MnwELRq/aVYCGg== 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.1] 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xl7cvlGxAAo/yapcUvrgkfnjIQVFODeg== 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.2] 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xlUs5dIRAA6O3abLCmXtpP5DrInddy2g== 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.3] 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xlrcPZJhAAzTHuhlQkgedj41V2tolmBg== 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.4] 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xl/uRVLBAAPVruNef6exPn4DIqlE2a3w== 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.5] 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xl2pLQMRAAVcplpQaS+vOg1Oyr0T7xAg== 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.6] 2023-12-27T21:48:33.914 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQClm4xllxJQNxAA1tFSpOlHglRwbdT9UXRzcA== 2023-12-27T21:48:33.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-27T21:48:34.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-27T21:48:34.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-27T21:48:34.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-27T21:48:34.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:48:34.612 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-27T21:48:34.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-27T21:48:34.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-27T21:48:34.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-27T21:48:34.757 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-27T21:48:34.814 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-27T21:48:34.815 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-27T21:48:34.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-27T21:48:34.857 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-27T21:48:34.857 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-27T21:48:34.857 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-27T21:48:34.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-27T21:48:34.859 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-27T21:48:34.859 INFO:tasks.ceph:Setting crush tunables to default 2023-12-27T21:48:34.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-27T21:48:35.080 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2023-12-27T21:48:35.091 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-27T21:48:35.091 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:35.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-27T21:48:35.117 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:35.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-27T21:48:35.184 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:35.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-27T21:48:35.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster cluster1 osd new d24478b2-7bf5-4df4-b108-249a106ca9f8 0 2023-12-27T21:48:35.267 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:35.267+0000 7fbda91e6680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-27T21:48:35.343 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:35.343+0000 7fbda91e6680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-27T21:48:35.474 INFO:teuthology.orchestra.run.smithi086.stdout:0 2023-12-27T21:48:35.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster cluster1 osd new 19dcf9d9-78ab-4746-ab3f-574f132b3148 1 2023-12-27T21:48:35.696 INFO:teuthology.orchestra.run.smithi086.stdout:1 2023-12-27T21:48:35.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster cluster1 osd new 5895f6a1-4f3e-4390-a5ca-1138e24b2cc1 2 2023-12-27T21:48:35.776 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:35.776+0000 7fbda91e6680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-27T21:48:35.914 INFO:teuthology.orchestra.run.smithi086.stdout:2 2023-12-27T21:48:35.927 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-27T21:48:35.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-27T21:48:35.932 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-27T21:48:35.932 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-27T21:48:35.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-27T21:48:35.939 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-27T21:48:35.939 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-27T21:48:35.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-27T21:48:35.943 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-27T21:48:35.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-27T21:48:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:36.166 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":5,"fsid":"b14702b5-3263-4304-8c6f-fcd3bab3acd5","created":"2023-12-27T21:48:34.990755+0000","modified":"2023-12-27T21:48:35.914062+0000","last_up_change":"0.000000","last_in_change":"2023-12-27T21:48:35.914062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"d24478b2-7bf5-4df4-b108-249a106ca9f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"19dcf9d9-78ab-4746-ab3f-574f132b3148","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5895f6a1-4f3e-4390-a5ca-1138e24b2cc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:48:36.178 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-27T21:48:36.179 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-27T21:48:36.481 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-27T21:48:36.536 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:36.536+0000 7fbda91e6680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-27T21:48:36.536 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T21:48:36.536+0000 7f8298349380 -1 Falling back to public interface 2023-12-27T21:48:36.544 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T21:48:36.544+0000 7fa0f231c380 -1 Falling back to public interface 2023-12-27T21:48:36.556 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T21:48:36.556+0000 7f67fdc8e380 -1 Falling back to public interface 2023-12-27T21:48:36.607 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:36.607+0000 7fbda91e6680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-27T21:48:36.793 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:36.794+0000 7fbda91e6680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-27T21:48:36.885 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-27T21:48:36.885 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":5,"fsid":"b14702b5-3263-4304-8c6f-fcd3bab3acd5","created":"2023-12-27T21:48:34.990755+0000","modified":"2023-12-27T21:48:35.914062+0000","last_up_change":"0.000000","last_in_change":"2023-12-27T21:48:35.914062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"d24478b2-7bf5-4df4-b108-249a106ca9f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"19dcf9d9-78ab-4746-ab3f-574f132b3148","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5895f6a1-4f3e-4390-a5ca-1138e24b2cc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:48:36.896 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-27T21:48:37.235 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:37.235+0000 7fbda91e6680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-27T21:48:37.337 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:37.337+0000 7fbda91e6680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-27T21:48:37.490 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:37.491+0000 7fbda91e6680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-27T21:48:37.567 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:37.567+0000 7fbda91e6680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-27T21:48:37.636 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:37.636+0000 7fbda91e6680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-27T21:48:37.897 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:37.897+0000 7fbda91e6680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-27T21:48:38.091 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:38.091+0000 7fbda91e6680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-27T21:48:38.757 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:38.757+0000 7fbda91e6680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-27T21:48:38.827 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:38.827+0000 7fbda91e6680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-27T21:48:38.905 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:38.906+0000 7fbda91e6680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-27T21:48:39.055 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:39.055+0000 7fbda91e6680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-27T21:48:39.131 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:39.131+0000 7fbda91e6680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-27T21:48:39.247 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:39.247+0000 7fbda91e6680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-27T21:48:39.377 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:39.377+0000 7fbda91e6680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-27T21:48:39.723 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:39.723+0000 7fbda91e6680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-27T21:48:39.793 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:39.793+0000 7fbda91e6680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-27T21:48:40.096 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T21:48:40.096+0000 7f67fdc8e380 -1 osd.2 0 log_to_monitors {default=true} 2023-12-27T21:48:40.121 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T21:48:40.122+0000 7f8298349380 -1 osd.0 0 log_to_monitors {default=true} 2023-12-27T21:48:40.122 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T21:48:40.123+0000 7fa0f231c380 -1 osd.1 0 log_to_monitors {default=true} 2023-12-27T21:48:40.812 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T21:48:40.812+0000 7fbd5ea27700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-27T21:48:41.951 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T21:48:41.952+0000 7f67f4645700 -1 osd.2 0 waiting for initial osdmap 2023-12-27T21:48:41.952 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T21:48:41.952+0000 7fa0e8cd3700 -1 osd.1 0 waiting for initial osdmap 2023-12-27T21:48:41.952 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T21:48:41.952+0000 7f828ed00700 -1 osd.0 0 waiting for initial osdmap 2023-12-27T21:48:41.956 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T21:48:41.956+0000 7f828b47c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-27T21:48:41.956 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T21:48:41.956+0000 7fa0e544f700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-27T21:48:41.956 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T21:48:41.956+0000 7f67f0dc1700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-27T21:48:43.199 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-27T21:48:43.490 INFO:teuthology.misc.health.smithi086.stdout: 2023-12-27T21:48:43.490 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":8,"fsid":"b14702b5-3263-4304-8c6f-fcd3bab3acd5","created":"2023-12-27T21:48:34.990755+0000","modified":"2023-12-27T21:48:42.947801+0000","last_up_change":"2023-12-27T21:48:42.947801+0000","last_in_change":"2023-12-27T21:48:35.914062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"d24478b2-7bf5-4df4-b108-249a106ca9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41906},{"type":"v1","addr":"172.21.15.86:6801","nonce":41906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":41906},{"type":"v1","addr":"172.21.15.86:6803","nonce":41906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":41906},{"type":"v1","addr":"172.21.15.86:6807","nonce":41906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41906},{"type":"v1","addr":"172.21.15.86:6805","nonce":41906}]},"public_addr":"172.21.15.86:6801/41906","cluster_addr":"172.21.15.86:6803/41906","heartbeat_back_addr":"172.21.15.86:6807/41906","heartbeat_front_addr":"172.21.15.86:6805/41906","state":["exists","up"]},{"osd":1,"uuid":"19dcf9d9-78ab-4746-ab3f-574f132b3148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41907},{"type":"v1","addr":"172.21.15.86:6809","nonce":41907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":41907},{"type":"v1","addr":"172.21.15.86:6811","nonce":41907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":41907},{"type":"v1","addr":"172.21.15.86:6815","nonce":41907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41907},{"type":"v1","addr":"172.21.15.86:6813","nonce":41907}]},"public_addr":"172.21.15.86:6809/41907","cluster_addr":"172.21.15.86:6811/41907","heartbeat_back_addr":"172.21.15.86:6815/41907","heartbeat_front_addr":"172.21.15.86:6813/41907","state":["exists","up"]},{"osd":2,"uuid":"5895f6a1-4f3e-4390-a5ca-1138e24b2cc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41930},{"type":"v1","addr":"172.21.15.86:6817","nonce":41930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41930},{"type":"v1","addr":"172.21.15.86:6819","nonce":41930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41930},{"type":"v1","addr":"172.21.15.86:6823","nonce":41930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41930},{"type":"v1","addr":"172.21.15.86:6821","nonce":41930}]},"public_addr":"172.21.15.86:6817/41930","cluster_addr":"172.21.15.86:6819/41930","heartbeat_back_addr":"172.21.15.86:6823/41930","heartbeat_front_addr":"172.21.15.86:6821/41930","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:48:43.502 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-27T21:48:43.503 INFO:tasks.ceph:Creating RBD pool 2023-12-27T21:48:43.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-27T21:48:43.964 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'rbd' created 2023-12-27T21:48:43.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-27T21:48:44.968 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-27T21:48:44.982 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-27T21:48:44.983 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-27T21:48:44.983 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-27T21:48:44.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-27T21:48:44.983 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-27T21:48:45.318 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:45.331 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":41529},{"type":"v1","addr":"172.21.15.86:6825","nonce":41529}]},"active_addr":"172.21.15.86:6825/41529","active_change":"2023-12-27T21:48:39.807869+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3620247003}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2003435205}]}]} 2023-12-27T21:48:45.333 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-27T21:48:45.333 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-27T21:48:45.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-27T21:48:45.621 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:45.621 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":10,"fsid":"b14702b5-3263-4304-8c6f-fcd3bab3acd5","created":"2023-12-27T21:48:34.990755+0000","modified":"2023-12-27T21:48:44.958761+0000","last_up_change":"2023-12-27T21:48:42.947801+0000","last_in_change":"2023-12-27T21:48:35.914062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-27T21:48:42.987515+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-27T21:48:43.823029+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d24478b2-7bf5-4df4-b108-249a106ca9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41906},{"type":"v1","addr":"172.21.15.86:6801","nonce":41906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":41906},{"type":"v1","addr":"172.21.15.86:6803","nonce":41906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":41906},{"type":"v1","addr":"172.21.15.86:6807","nonce":41906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41906},{"type":"v1","addr":"172.21.15.86:6805","nonce":41906}]},"public_addr":"172.21.15.86:6801/41906","cluster_addr":"172.21.15.86:6803/41906","heartbeat_back_addr":"172.21.15.86:6807/41906","heartbeat_front_addr":"172.21.15.86:6805/41906","state":["exists","up"]},{"osd":1,"uuid":"19dcf9d9-78ab-4746-ab3f-574f132b3148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41907},{"type":"v1","addr":"172.21.15.86:6809","nonce":41907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":41907},{"type":"v1","addr":"172.21.15.86:6811","nonce":41907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":41907},{"type":"v1","addr":"172.21.15.86:6815","nonce":41907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41907},{"type":"v1","addr":"172.21.15.86:6813","nonce":41907}]},"public_addr":"172.21.15.86:6809/41907","cluster_addr":"172.21.15.86:6811/41907","heartbeat_back_addr":"172.21.15.86:6815/41907","heartbeat_front_addr":"172.21.15.86:6813/41907","state":["exists","up"]},{"osd":2,"uuid":"5895f6a1-4f3e-4390-a5ca-1138e24b2cc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41930},{"type":"v1","addr":"172.21.15.86:6817","nonce":41930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41930},{"type":"v1","addr":"172.21.15.86:6819","nonce":41930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41930},{"type":"v1","addr":"172.21.15.86:6823","nonce":41930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41930},{"type":"v1","addr":"172.21.15.86:6821","nonce":41930}]},"public_addr":"172.21.15.86:6817/41930","cluster_addr":"172.21.15.86:6819/41930","heartbeat_back_addr":"172.21.15.86:6823/41930","heartbeat_front_addr":"172.21.15.86:6821/41930","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:48:41.122710+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:48:41.086453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:48:41.062866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:48:45.634 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-27T21:48:45.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-27T21:48:45.924 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:45.924 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":10,"fsid":"b14702b5-3263-4304-8c6f-fcd3bab3acd5","created":"2023-12-27T21:48:34.990755+0000","modified":"2023-12-27T21:48:44.958761+0000","last_up_change":"2023-12-27T21:48:42.947801+0000","last_in_change":"2023-12-27T21:48:35.914062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-27T21:48:42.987515+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-27T21:48:43.823029+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d24478b2-7bf5-4df4-b108-249a106ca9f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":41906},{"type":"v1","addr":"172.21.15.86:6801","nonce":41906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":41906},{"type":"v1","addr":"172.21.15.86:6803","nonce":41906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":41906},{"type":"v1","addr":"172.21.15.86:6807","nonce":41906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":41906},{"type":"v1","addr":"172.21.15.86:6805","nonce":41906}]},"public_addr":"172.21.15.86:6801/41906","cluster_addr":"172.21.15.86:6803/41906","heartbeat_back_addr":"172.21.15.86:6807/41906","heartbeat_front_addr":"172.21.15.86:6805/41906","state":["exists","up"]},{"osd":1,"uuid":"19dcf9d9-78ab-4746-ab3f-574f132b3148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":41907},{"type":"v1","addr":"172.21.15.86:6809","nonce":41907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":41907},{"type":"v1","addr":"172.21.15.86:6811","nonce":41907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":41907},{"type":"v1","addr":"172.21.15.86:6815","nonce":41907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":41907},{"type":"v1","addr":"172.21.15.86:6813","nonce":41907}]},"public_addr":"172.21.15.86:6809/41907","cluster_addr":"172.21.15.86:6811/41907","heartbeat_back_addr":"172.21.15.86:6815/41907","heartbeat_front_addr":"172.21.15.86:6813/41907","state":["exists","up"]},{"osd":2,"uuid":"5895f6a1-4f3e-4390-a5ca-1138e24b2cc1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":41930},{"type":"v1","addr":"172.21.15.86:6817","nonce":41930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":41930},{"type":"v1","addr":"172.21.15.86:6819","nonce":41930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":41930},{"type":"v1","addr":"172.21.15.86:6823","nonce":41930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":41930},{"type":"v1","addr":"172.21.15.86:6821","nonce":41930}]},"public_addr":"172.21.15.86:6817/41930","cluster_addr":"172.21.15.86:6819/41930","heartbeat_back_addr":"172.21.15.86:6823/41930","heartbeat_front_addr":"172.21.15.86:6821/41930","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:48:41.122710+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:48:41.086453+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:48:41.062866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:48:45.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-27T21:48:46.087 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-12-27T21:48:46.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-27T21:48:46.236 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-12-27T21:48:46.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-27T21:48:46.384 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-12-27T21:48:46.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-27T21:48:46.678 INFO:teuthology.orchestra.run.smithi086.stdout:34359738370 2023-12-27T21:48:46.691 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2023-12-27T21:48:47.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-27T21:48:47.997 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-12-27T21:48:48.011 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2023-12-27T21:48:48.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-27T21:48:48.301 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-12-27T21:48:48.314 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2023-12-27T21:48:48.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-27T21:48:48.602 INFO:teuthology.orchestra.run.smithi086.stdout:34359738371 2023-12-27T21:48:48.616 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2023-12-27T21:48:48.616 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-27T21:48:48.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-27T21:48:48.903 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:48.905 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-27T21:48:48.917 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-12-27T21:48:47.812524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223960,"statfs":{"total":289910292480,"available":288997335040,"internally_reserved":0,"allocated":278528,"data_stored":72600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.975394+0000","last_change":"2023-12-27T21:48:44.974525+0000","last_active":"2023-12-27T21:48:44.975394+0000","last_peered":"2023-12-27T21:48:44.975394+0000","last_clean":"2023-12-27T21:48:44.975394+0000","last_became_active":"2023-12-27T21:48:44.974224+0000","last_became_peered":"2023-12-27T21:48:44.974224+0000","last_unstale":"2023-12-27T21:48:44.975394+0000","last_undegraded":"2023-12-27T21:48:44.975394+0000","last_fullsized":"2023-12-27T21:48:44.975394+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973486+0000","last_change":"2023-12-27T21:48:44.971409+0000","last_active":"2023-12-27T21:48:44.973486+0000","last_peered":"2023-12-27T21:48:44.973486+0000","last_clean":"2023-12-27T21:48:44.973486+0000","last_became_active":"2023-12-27T21:48:44.971194+0000","last_became_peered":"2023-12-27T21:48:44.971194+0000","last_unstale":"2023-12-27T21:48:44.973486+0000","last_undegraded":"2023-12-27T21:48:44.973486+0000","last_fullsized":"2023-12-27T21:48:44.973486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.975388+0000","last_change":"2023-12-27T21:48:44.974456+0000","last_active":"2023-12-27T21:48:44.975388+0000","last_peered":"2023-12-27T21:48:44.975388+0000","last_clean":"2023-12-27T21:48:44.975388+0000","last_became_active":"2023-12-27T21:48:44.974198+0000","last_became_peered":"2023-12-27T21:48:44.974198+0000","last_unstale":"2023-12-27T21:48:44.975388+0000","last_undegraded":"2023-12-27T21:48:44.975388+0000","last_fullsized":"2023-12-27T21:48:44.975388+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973226+0000","last_change":"2023-12-27T21:48:44.973226+0000","last_active":"2023-12-27T21:48:44.973226+0000","last_peered":"2023-12-27T21:48:44.973226+0000","last_clean":"2023-12-27T21:48:44.973226+0000","last_became_active":"2023-12-27T21:48:44.972954+0000","last_became_peered":"2023-12-27T21:48:44.972954+0000","last_unstale":"2023-12-27T21:48:44.973226+0000","last_undegraded":"2023-12-27T21:48:44.973226+0000","last_fullsized":"2023-12-27T21:48:44.973226+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973537+0000","last_change":"2023-12-27T21:48:44.971537+0000","last_active":"2023-12-27T21:48:44.973537+0000","last_peered":"2023-12-27T21:48:44.973537+0000","last_clean":"2023-12-27T21:48:44.973537+0000","last_became_active":"2023-12-27T21:48:44.971253+0000","last_became_peered":"2023-12-27T21:48:44.971253+0000","last_unstale":"2023-12-27T21:48:44.973537+0000","last_undegraded":"2023-12-27T21:48:44.973537+0000","last_fullsized":"2023-12-27T21:48:44.973537+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.974011+0000","last_change":"2023-12-27T21:48:44.972743+0000","last_active":"2023-12-27T21:48:44.974011+0000","last_peered":"2023-12-27T21:48:44.974011+0000","last_clean":"2023-12-27T21:48:44.974011+0000","last_became_active":"2023-12-27T21:48:44.972464+0000","last_became_peered":"2023-12-27T21:48:44.972464+0000","last_unstale":"2023-12-27T21:48:44.974011+0000","last_undegraded":"2023-12-27T21:48:44.974011+0000","last_fullsized":"2023-12-27T21:48:44.974011+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973074+0000","last_change":"2023-12-27T21:48:44.973074+0000","last_active":"2023-12-27T21:48:44.973074+0000","last_peered":"2023-12-27T21:48:44.973074+0000","last_clean":"2023-12-27T21:48:44.973074+0000","last_became_active":"2023-12-27T21:48:44.972792+0000","last_became_peered":"2023-12-27T21:48:44.972792+0000","last_unstale":"2023-12-27T21:48:44.973074+0000","last_undegraded":"2023-12-27T21:48:44.973074+0000","last_fullsized":"2023-12-27T21:48:44.973074+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.971780+0000","last_change":"2023-12-27T21:48:44.971780+0000","last_active":"2023-12-27T21:48:44.971780+0000","last_peered":"2023-12-27T21:48:44.971780+0000","last_clean":"2023-12-27T21:48:44.971780+0000","last_became_active":"2023-12-27T21:48:44.971479+0000","last_became_peered":"2023-12-27T21:48:44.971479+0000","last_unstale":"2023-12-27T21:48:44.971780+0000","last_undegraded":"2023-12-27T21:48:44.971780+0000","last_fullsized":"2023-12-27T21:48:44.971780+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.975026+0000","last_change":"2023-12-27T21:48:44.973108+0000","last_active":"2023-12-27T21:48:44.975026+0000","last_peered":"2023-12-27T21:48:44.975026+0000","last_clean":"2023-12-27T21:48:44.975026+0000","last_became_active":"2023-12-27T21:48:44.972769+0000","last_became_peered":"2023-12-27T21:48:44.972769+0000","last_unstale":"2023-12-27T21:48:44.975026+0000","last_undegraded":"2023-12-27T21:48:44.975026+0000","last_fullsized":"2023-12-27T21:48:44.975026+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-27T21:48:48.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-27T21:48:49.202 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2023-12-27T21:48:49.215 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-12-27T21:48:47.812524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223960,"statfs":{"total":289910292480,"available":288997335040,"internally_reserved":0,"allocated":278528,"data_stored":72600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.975394+0000","last_change":"2023-12-27T21:48:44.974525+0000","last_active":"2023-12-27T21:48:44.975394+0000","last_peered":"2023-12-27T21:48:44.975394+0000","last_clean":"2023-12-27T21:48:44.975394+0000","last_became_active":"2023-12-27T21:48:44.974224+0000","last_became_peered":"2023-12-27T21:48:44.974224+0000","last_unstale":"2023-12-27T21:48:44.975394+0000","last_undegraded":"2023-12-27T21:48:44.975394+0000","last_fullsized":"2023-12-27T21:48:44.975394+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973486+0000","last_change":"2023-12-27T21:48:44.971409+0000","last_active":"2023-12-27T21:48:44.973486+0000","last_peered":"2023-12-27T21:48:44.973486+0000","last_clean":"2023-12-27T21:48:44.973486+0000","last_became_active":"2023-12-27T21:48:44.971194+0000","last_became_peered":"2023-12-27T21:48:44.971194+0000","last_unstale":"2023-12-27T21:48:44.973486+0000","last_undegraded":"2023-12-27T21:48:44.973486+0000","last_fullsized":"2023-12-27T21:48:44.973486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.975388+0000","last_change":"2023-12-27T21:48:44.974456+0000","last_active":"2023-12-27T21:48:44.975388+0000","last_peered":"2023-12-27T21:48:44.975388+0000","last_clean":"2023-12-27T21:48:44.975388+0000","last_became_active":"2023-12-27T21:48:44.974198+0000","last_became_peered":"2023-12-27T21:48:44.974198+0000","last_unstale":"2023-12-27T21:48:44.975388+0000","last_undegraded":"2023-12-27T21:48:44.975388+0000","last_fullsized":"2023-12-27T21:48:44.975388+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973226+0000","last_change":"2023-12-27T21:48:44.973226+0000","last_active":"2023-12-27T21:48:44.973226+0000","last_peered":"2023-12-27T21:48:44.973226+0000","last_clean":"2023-12-27T21:48:44.973226+0000","last_became_active":"2023-12-27T21:48:44.972954+0000","last_became_peered":"2023-12-27T21:48:44.972954+0000","last_unstale":"2023-12-27T21:48:44.973226+0000","last_undegraded":"2023-12-27T21:48:44.973226+0000","last_fullsized":"2023-12-27T21:48:44.973226+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973537+0000","last_change":"2023-12-27T21:48:44.971537+0000","last_active":"2023-12-27T21:48:44.973537+0000","last_peered":"2023-12-27T21:48:44.973537+0000","last_clean":"2023-12-27T21:48:44.973537+0000","last_became_active":"2023-12-27T21:48:44.971253+0000","last_became_peered":"2023-12-27T21:48:44.971253+0000","last_unstale":"2023-12-27T21:48:44.973537+0000","last_undegraded":"2023-12-27T21:48:44.973537+0000","last_fullsized":"2023-12-27T21:48:44.973537+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.974011+0000","last_change":"2023-12-27T21:48:44.972743+0000","last_active":"2023-12-27T21:48:44.974011+0000","last_peered":"2023-12-27T21:48:44.974011+0000","last_clean":"2023-12-27T21:48:44.974011+0000","last_became_active":"2023-12-27T21:48:44.972464+0000","last_became_peered":"2023-12-27T21:48:44.972464+0000","last_unstale":"2023-12-27T21:48:44.974011+0000","last_undegraded":"2023-12-27T21:48:44.974011+0000","last_fullsized":"2023-12-27T21:48:44.974011+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.973074+0000","last_change":"2023-12-27T21:48:44.973074+0000","last_active":"2023-12-27T21:48:44.973074+0000","last_peered":"2023-12-27T21:48:44.973074+0000","last_clean":"2023-12-27T21:48:44.973074+0000","last_became_active":"2023-12-27T21:48:44.972792+0000","last_became_peered":"2023-12-27T21:48:44.972792+0000","last_unstale":"2023-12-27T21:48:44.973074+0000","last_undegraded":"2023-12-27T21:48:44.973074+0000","last_fullsized":"2023-12-27T21:48:44.973074+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.971780+0000","last_change":"2023-12-27T21:48:44.971780+0000","last_active":"2023-12-27T21:48:44.971780+0000","last_peered":"2023-12-27T21:48:44.971780+0000","last_clean":"2023-12-27T21:48:44.971780+0000","last_became_active":"2023-12-27T21:48:44.971479+0000","last_became_peered":"2023-12-27T21:48:44.971479+0000","last_unstale":"2023-12-27T21:48:44.971780+0000","last_undegraded":"2023-12-27T21:48:44.971780+0000","last_fullsized":"2023-12-27T21:48:44.971780+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:48:44.975026+0000","last_change":"2023-12-27T21:48:44.973108+0000","last_active":"2023-12-27T21:48:44.975026+0000","last_peered":"2023-12-27T21:48:44.975026+0000","last_clean":"2023-12-27T21:48:44.975026+0000","last_became_active":"2023-12-27T21:48:44.972769+0000","last_became_peered":"2023-12-27T21:48:44.972769+0000","last_unstale":"2023-12-27T21:48:44.975026+0000","last_undegraded":"2023-12-27T21:48:44.975026+0000","last_fullsized":"2023-12-27T21:48:44.975026+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:48:43.952605+0000","last_clean_scrub_stamp":"2023-12-27T21:48:43.952605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-27T21:48:49.216 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-27T21:48:49.216 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-27T21:48:49.216 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-27T21:48:49.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-27T21:48:49.540 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:49.553 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-27T21:48:49.553 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-27T21:48:49.554 INFO:teuthology.run_tasks:Running task ceph... 2023-12-27T21:48:49.563 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-27T21:48:49.563 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-27T21:48:49.563 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi/7502301', 'branch': 'wip-yuri7-testing-2023-12-27-1008-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7502301', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-27_20:59:12-rbd-wip-yuri7-testing-2023-12-27-1008-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2023-12-27-1008-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri7-testing-2023-12-27-1008-pacific', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5573, 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri7-testing-2023-12-27-1008-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d', 'targets': {'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOblRzH4CISFZgKXd44gHxbtkO20DUUrc3Tk8ZWISfJl0F8R7ZnFxuu/tw3Bx18hMtCqyI+gKAQQNinSK5R6+80=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQEeKbf/5+MFgp1CU514Rt/z8u7LW2e6ayiIfhvGzRKJnbTd60HsQS9USBrvGz6/7ODtWja1GHRU4zoLdGn7aU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c5564c7988cbaadc3382253af5843a8595347c2d'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '01dd171a561ca33631bf0a38b105879779f72410', 'timestamp': '2023-12-27_20:59:12', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-27T21:48:49.564 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-27T21:48:49.566 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-27T21:48:49.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2023-12-27T21:48:49.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2023-12-27T21:48:49.658 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:49.658 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2023-12-27T21:48:49.718 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-27T21:48:49.718 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2023-12-27T21:48:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-27T21:48:49.776 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:49.777 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 125492 Links: 1 2023-12-27T21:48:49.777 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:49.777 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:49.777 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-12-27 21:47:33.330764403 +0000 2023-12-27T21:48:49.777 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-12-27 21:38:35.119461107 +0000 2023-12-27T21:48:49.777 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-12-27 21:38:35.119461107 +0000 2023-12-27T21:48:49.777 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-12-27T21:48:49.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-27T21:48:49.844 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-12-27T21:48:49.845 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-12-27T21:48:49.845 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000277824 s, 1.8 MB/s 2023-12-27T21:48:49.849 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-27T21:48:49.907 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2023-12-27T21:48:49.965 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-27T21:48:49.965 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:49.966 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 117486 Links: 1 2023-12-27T21:48:49.966 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:49.966 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:49.966 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-12-27 21:47:33.330764403 +0000 2023-12-27T21:48:49.966 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-12-27 21:38:35.616450201 +0000 2023-12-27T21:48:49.966 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-12-27 21:38:35.616450201 +0000 2023-12-27T21:48:49.966 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-12-27T21:48:49.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-27T21:48:50.032 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-12-27T21:48:50.032 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-12-27T21:48:50.033 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000322175 s, 1.6 MB/s 2023-12-27T21:48:50.034 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-27T21:48:50.091 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 128382 Links: 1 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-12-27 21:47:33.330764403 +0000 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-12-27 21:38:36.103439514 +0000 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-12-27 21:38:36.103439514 +0000 2023-12-27T21:48:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-12-27T21:48:50.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-27T21:48:50.214 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-12-27T21:48:50.214 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-12-27T21:48:50.215 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000244172 s, 2.1 MB/s 2023-12-27T21:48:50.215 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-27T21:48:50.273 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2023-12-27T21:48:50.330 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 127533 Links: 1 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-12-27 21:47:33.330764403 +0000 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-12-27 21:38:36.601428586 +0000 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-12-27 21:38:36.601428586 +0000 2023-12-27T21:48:50.331 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-12-27T21:48:50.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-27T21:48:50.396 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-12-27T21:48:50.396 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-12-27T21:48:50.396 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000241124 s, 2.1 MB/s 2023-12-27T21:48:50.397 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-27T21:48:50.454 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:50.454 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-27T21:48:50.454 INFO:tasks.ceph:Generating config... 2023-12-27T21:48:50.455 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client] log to stderr = False 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.456 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.456 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-27T21:48:50.457 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-27T21:48:50.457 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-27T21:48:50.457 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-27T21:48:50.457 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-27T21:48:50.457 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-27T21:48:50.457 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-27T21:48:50.457 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-27T21:48:50.458 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-27T21:48:50.458 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-27T21:48:50.458 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-27T21:48:50.458 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-27T21:48:50.458 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-27T21:48:50.458 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-27T21:48:50.458 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-27T21:48:50.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-27T21:48:50.533 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.keyring 2023-12-27T21:48:50.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-27T21:48:50.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-27T21:48:50.684 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.196')] 2023-12-27T21:48:50.684 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.196', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-27T21:48:50.685 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:50.685 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-27T21:48:50.740 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.196 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:monmaptool: generated fsid c088ab28-4839-43bd-96eb-95acf66420e1 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:epoch 0 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:fsid c088ab28-4839-43bd-96eb-95acf66420e1 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:last_changed 2023-12-27T21:48:50.819379+0000 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:created 2023-12-27T21:48:50.819379+0000 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:min_mon_release 0 (unknown) 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:election_strategy: 1 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:0: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.a 2023-12-27T21:48:50.818 INFO:teuthology.orchestra.run.smithi196.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-27T21:48:50.819 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-27T21:48:50.877 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID c088ab28-4839-43bd-96eb-95acf66420e1... 2023-12-27T21:48:50.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-27T21:48:50.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-27T21:48:50.932 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.933 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2023-12-27T21:48:50.934 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = 172.21.15.196 2023-12-27T21:48:50.935 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 5000 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = c088ab28-4839-43bd-96eb-95acf66420e1 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2023-12-27T21:48:50.936 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2023-12-27T21:48:50.937 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: bdev async discard = True 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: bdev enable discard = True 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore allocator = stupid 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 1/20 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 1/20 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 4/10 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2023-12-27T21:48:50.938 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2023-12-27T21:48:50.939 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2023-12-27T21:48:50.940 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: debug rbd = 20 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: debug rbd_mirror = 15 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: log to stderr = False 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: rbd default features = 125 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: rbd mirror image policy type = none 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.0] 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.1] 2023-12-27T21:48:50.941 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.942 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.942 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.2] 2023-12-27T21:48:50.942 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.942 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.942 INFO:teuthology.orchestra.run.smithi086.stdout:[client.mirror.3] 2023-12-27T21:48:50.942 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.942 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.962 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2023-12-27T21:48:50.962 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2023-12-27T21:48:50.963 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-27T21:48:50.964 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = 172.21.15.196 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: mon client directed command retry = 5 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: ms inject socket failures = 5000 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = c088ab28-4839-43bd-96eb-95acf66420e1 2023-12-27T21:48:50.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2023-12-27T21:48:50.966 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: bdev async discard = True 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: bdev enable discard = True 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore allocator = stupid 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2023-12-27T21:48:50.967 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 1/20 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 1/20 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 4/10 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2023-12-27T21:48:50.968 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2023-12-27T21:48:50.969 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2023-12-27T21:48:50.970 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: debug rbd = 20 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: debug rbd_mirror = 15 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: log to stderr = False 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: rbd default features = 125 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: rbd mirror image policy type = none 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.0] 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.1] 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.2] 2023-12-27T21:48:50.971 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.972 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.972 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.3] 2023-12-27T21:48:50.972 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-27T21:48:50.972 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-27T21:48:50.974 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-27T21:48:50.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-27T21:48:51.025 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-27T21:48:51.025 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:51.026 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-27T21:48:51.042 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:51.042 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-27T21:48:51.099 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2023-12-27T21:48:51.100 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:51.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-27T21:48:51.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-27T21:48:51.142 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:48:51.142 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:48:51.199 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2023-12-27T21:48:51.199 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:51.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-27T21:48:51.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-27T21:48:51.236 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:48:51.236 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:48:51.294 INFO:tasks.ceph:Setting up mon nodes... 2023-12-27T21:48:51.294 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-27T21:48:51.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-27T21:48:51.347 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-27T21:48:51.349 INFO:tasks.ceph:Setting up mds nodes... 2023-12-27T21:48:51.349 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-27T21:48:51.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-27T21:48:51.389 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-27T21:48:51.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-27T21:48:51.441 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-27T21:48:51.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-27T21:48:51.533 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-27T21:48:51.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-27T21:48:51.625 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-27T21:48:51.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-27T21:48:51.717 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-27T21:48:51.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-27T21:48:51.809 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-27T21:48:51.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-27T21:48:51.901 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-27T21:48:51.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-27T21:48:51.993 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-27T21:48:52.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-27T21:48:52.085 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-27T21:48:52.098 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-27T21:48:52.099 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-27T21:48:52.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-27T21:48:52.165 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:52.166 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-27T21:48:52.166 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2023-12-27T21:48:52.166 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-27T21:48:52.240 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-27T21:48:52.240 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-27T21:48:52.240 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-27T21:48:52.240 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-27T21:48:52.240 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-27T21:48:52.240 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2023-12-27T21:48:52.240 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-27T21:48:52.241 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-27T21:48:52.241 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-27T21:48:52.241 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-27T21:48:52.676 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2023-12-27T21:48:52.678 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2023-12-27T21:48:52.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-27T21:48:52.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-27T21:48:52.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-27T21:48:52.958 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:52.958 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-27T21:48:52.958 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2023-12-27T21:48:52.958 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-27T21:48:53.033 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-27T21:48:53.033 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-27T21:48:53.033 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-27T21:48:53.033 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-27T21:48:53.033 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-27T21:48:53.033 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2023-12-27T21:48:53.033 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-27T21:48:53.034 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-27T21:48:53.034 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-27T21:48:53.034 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-27T21:48:53.454 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2023-12-27T21:48:53.456 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2023-12-27T21:48:53.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-27T21:48:53.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-27T21:48:53.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-27T21:48:53.661 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-27T21:48:53.661 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-27T21:48:53.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2023-12-27T21:48:53.661 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-27T21:48:53.734 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-27T21:48:53.735 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-27T21:48:53.735 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-27T21:48:54.182 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2023-12-27T21:48:54.185 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2023-12-27T21:48:54.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-27T21:48:54.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-27T21:48:54.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:48:54.477 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:54.478+0000 7f0278764380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-27T21:48:54.479 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:54.480+0000 7f0278764380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-27T21:48:54.479 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:54.480+0000 7f0278764380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-27T21:48:54.479 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:54.480+0000 7f0278764380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-27T21:48:54.479 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:54.480+0000 7f0278764380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-27T21:48:54.487 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:54.488+0000 7f0278764380 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-27T21:48:57.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-27T21:48:57.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:48:57.464 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:57.463+0000 7f7376187380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-27T21:48:57.464 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:57.463+0000 7f7376187380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-27T21:48:57.464 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:57.463+0000 7f7376187380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-27T21:48:57.464 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:57.463+0000 7f7376187380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-27T21:48:57.464 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:57.463+0000 7f7376187380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-27T21:48:57.464 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:48:57.464+0000 7f7376187380 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-27T21:49:00.310 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-27T21:49:00.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:49:00.425 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:49:00.424+0000 7fb9f9d8c380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-27T21:49:00.425 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:49:00.425+0000 7fb9f9d8c380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-27T21:49:00.425 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:49:00.425+0000 7fb9f9d8c380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-27T21:49:00.425 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:49:00.425+0000 7fb9f9d8c380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-27T21:49:00.425 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:49:00.425+0000 7fb9f9d8c380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-27T21:49:00.425 INFO:teuthology.orchestra.run.smithi196.stderr:2023-12-27T21:49:00.425+0000 7fb9f9d8c380 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-27T21:49:03.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-27T21:49:03.296 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-27T21:49:03.297 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:49:03.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-27T21:49:03.330 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.330 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-27T21:49:03.367 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-27T21:49:03.437 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-27T21:49:03.505 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-12-27T21:49:03.505 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-27T21:49:03.526 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.526 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-27T21:49:03.562 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.562 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-27T21:49:03.620 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.620 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-27T21:49:03.677 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.677 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-27T21:49:03.738 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.738 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-27T21:49:03.796 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.796 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-27T21:49:03.852 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.852 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-27T21:49:03.910 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:03.910 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-27T21:49:03.966 INFO:tasks.ceph:Adding keys to all mons... 2023-12-27T21:49:03.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-27T21:49:04.030 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2023-12-27T21:49:04.030 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xlsOnMFBAA2ZqdLVxECc+mWm05oQIcLw== 2023-12-27T21:49:04.030 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDGm4xlTQ6RHBAAqDclhAynJ+EqAGzjXyMh5g== 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDJm4xlErS3GxAATV1+XgJeV1ycwNXfMBKeKQ== 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDMm4xleIhoGRAAY7ciRZ7hrSX5boGczvz1Dg== 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xlzxBQFxAADgcNz4yXVuZ4UXo8JY1h8Q== 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror] 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xljSFkGhAAMy6fgfHT++e5YICQKzcrWg== 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.0] 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xl1pnhHxAA0rh/IzI3S06ewqSxbDUOkw== 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.1] 2023-12-27T21:49:04.031 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xliGZdJRAANYXe8RheWn3mEzh/rfv2QQ== 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.2] 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xlLsfUKhAAN3F4+nZzUvKQXyfihFfBMQ== 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.3] 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xlPJ1VMBAAjrZmwAyPJeISTNyLACBolQ== 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.4] 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xlAObUNRAAVh/H7IEhVAkR5G1UksqHcQ== 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.5] 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDDm4xlcQhNOxAAhe+TuOkq9eJp5pvaodfa4w== 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout:[client.mirror.6] 2023-12-27T21:49:04.032 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDEm4xl0v8wBRAAc3Cls6f1CvcncJFQzvyNgg== 2023-12-27T21:49:04.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-27T21:49:04.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-27T21:49:04.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-27T21:49:04.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-27T21:49:04.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-27T21:49:04.728 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-27T21:49:04.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-27T21:49:04.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-27T21:49:04.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-27T21:49:04.878 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-27T21:49:04.935 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-27T21:49:04.936 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-27T21:49:04.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-27T21:49:04.978 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-27T21:49:04.978 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-27T21:49:04.979 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-27T21:49:04.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-27T21:49:04.981 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-27T21:49:04.981 INFO:tasks.ceph:Setting crush tunables to default 2023-12-27T21:49:04.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-27T21:49:05.183 INFO:teuthology.orchestra.run.smithi196.stderr:adjusted tunables profile to default 2023-12-27T21:49:05.194 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-27T21:49:05.194 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:05.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-27T21:49:05.211 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:05.211+0000 7f7cf71a2680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-27T21:49:05.220 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:05.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-27T21:49:05.273 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:05.273+0000 7f7cf71a2680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-27T21:49:05.287 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:49:05.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-27T21:49:05.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster cluster2 osd new 7d01c453-d5a0-41a0-a464-d79d2985f9a9 0 2023-12-27T21:49:05.542 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:05.542+0000 7f7cf71a2680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-27T21:49:05.593 INFO:teuthology.orchestra.run.smithi196.stdout:0 2023-12-27T21:49:05.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster cluster2 osd new a16078f7-978e-40a5-8d09-00823ef94296 1 2023-12-27T21:49:05.803 INFO:teuthology.orchestra.run.smithi196.stdout:1 2023-12-27T21:49:05.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster cluster2 osd new 2ede6fbd-035a-4829-bb99-a202a0992f0c 2 2023-12-27T21:49:06.015 INFO:teuthology.orchestra.run.smithi196.stdout:2 2023-12-27T21:49:06.027 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-27T21:49:06.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-27T21:49:06.029 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-27T21:49:06.029 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-27T21:49:06.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-27T21:49:06.031 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-27T21:49:06.032 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-27T21:49:06.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-27T21:49:06.034 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-27T21:49:06.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-27T21:49:06.149 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:06.149+0000 7f7cf71a2680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-27T21:49:06.220 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:06.220+0000 7f7cf71a2680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-27T21:49:06.232 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:49:06.233 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":5,"fsid":"c088ab28-4839-43bd-96eb-95acf66420e1","created":"2023-12-27T21:49:05.091827+0000","modified":"2023-12-27T21:49:06.015055+0000","last_up_change":"0.000000","last_in_change":"2023-12-27T21:49:06.015055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7d01c453-d5a0-41a0-a464-d79d2985f9a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a16078f7-978e-40a5-8d09-00823ef94296","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2ede6fbd-035a-4829-bb99-a202a0992f0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:49:06.245 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-27T21:49:06.245 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-27T21:49:06.404 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:06.404+0000 7f7cf71a2680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-27T21:49:06.547 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-27T21:49:06.632 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T21:49:06.632+0000 7f7850b24380 -1 Falling back to public interface 2023-12-27T21:49:06.637 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T21:49:06.637+0000 7f4f4a48f380 -1 Falling back to public interface 2023-12-27T21:49:06.638 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T21:49:06.638+0000 7fc36d278380 -1 Falling back to public interface 2023-12-27T21:49:06.732 INFO:teuthology.misc.health.smithi196.stdout: 2023-12-27T21:49:06.733 INFO:teuthology.misc.health.smithi196.stdout:{"epoch":5,"fsid":"c088ab28-4839-43bd-96eb-95acf66420e1","created":"2023-12-27T21:49:05.091827+0000","modified":"2023-12-27T21:49:06.015055+0000","last_up_change":"0.000000","last_in_change":"2023-12-27T21:49:06.015055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7d01c453-d5a0-41a0-a464-d79d2985f9a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a16078f7-978e-40a5-8d09-00823ef94296","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2ede6fbd-035a-4829-bb99-a202a0992f0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:49:06.743 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-27T21:49:06.818 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:06.819+0000 7f7cf71a2680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-27T21:49:06.920 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:06.920+0000 7f7cf71a2680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-27T21:49:07.071 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:07.071+0000 7f7cf71a2680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-27T21:49:07.145 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:07.145+0000 7f7cf71a2680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-27T21:49:07.211 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:07.211+0000 7f7cf71a2680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-27T21:49:07.466 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:07.466+0000 7f7cf71a2680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-27T21:49:07.650 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:07.650+0000 7f7cf71a2680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-27T21:49:08.294 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:08.294+0000 7f7cf71a2680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-27T21:49:08.360 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:08.360+0000 7f7cf71a2680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-27T21:49:08.435 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:08.434+0000 7f7cf71a2680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-27T21:49:08.579 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:08.579+0000 7f7cf71a2680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-27T21:49:08.648 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:08.649+0000 7f7cf71a2680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-27T21:49:08.763 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:08.763+0000 7f7cf71a2680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-27T21:49:08.868 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:08.868+0000 7f7cf71a2680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-27T21:49:09.168 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:09.168+0000 7f7cf71a2680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-27T21:49:09.236 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:09.236+0000 7f7cf71a2680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-27T21:49:10.089 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T21:49:10.089+0000 7f7850b24380 -1 osd.0 0 log_to_monitors {default=true} 2023-12-27T21:49:10.138 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T21:49:10.138+0000 7f4f4a48f380 -1 osd.1 0 log_to_monitors {default=true} 2023-12-27T21:49:10.142 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T21:49:10.142+0000 7fc36d278380 -1 osd.2 0 log_to_monitors {default=true} 2023-12-27T21:49:10.256 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T21:49:10.256+0000 7f7caca44700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-27T21:49:12.072 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T21:49:12.072+0000 7f78474db700 -1 osd.0 0 waiting for initial osdmap 2023-12-27T21:49:12.073 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T21:49:12.072+0000 7f4f40e46700 -1 osd.1 0 waiting for initial osdmap 2023-12-27T21:49:12.073 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T21:49:12.072+0000 7fc363c2f700 -1 osd.2 0 waiting for initial osdmap 2023-12-27T21:49:12.076 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T21:49:12.076+0000 7f4f3d5c2700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-27T21:49:12.076 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T21:49:12.076+0000 7f7843c57700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-27T21:49:12.079 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T21:49:12.076+0000 7fc3603ab700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-27T21:49:13.046 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-27T21:49:13.344 INFO:teuthology.misc.health.smithi196.stdout: 2023-12-27T21:49:13.345 INFO:teuthology.misc.health.smithi196.stdout:{"epoch":8,"fsid":"c088ab28-4839-43bd-96eb-95acf66420e1","created":"2023-12-27T21:49:05.091827+0000","modified":"2023-12-27T21:49:13.068458+0000","last_up_change":"2023-12-27T21:49:13.068458+0000","last_in_change":"2023-12-27T21:49:06.015055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7d01c453-d5a0-41a0-a464-d79d2985f9a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":42641},{"type":"v1","addr":"172.21.15.196:6801","nonce":42641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":42641},{"type":"v1","addr":"172.21.15.196:6803","nonce":42641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":42641},{"type":"v1","addr":"172.21.15.196:6807","nonce":42641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":42641},{"type":"v1","addr":"172.21.15.196:6805","nonce":42641}]},"public_addr":"172.21.15.196:6801/42641","cluster_addr":"172.21.15.196:6803/42641","heartbeat_back_addr":"172.21.15.196:6807/42641","heartbeat_front_addr":"172.21.15.196:6805/42641","state":["exists","up"]},{"osd":1,"uuid":"a16078f7-978e-40a5-8d09-00823ef94296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":42642},{"type":"v1","addr":"172.21.15.196:6809","nonce":42642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":42642},{"type":"v1","addr":"172.21.15.196:6812","nonce":42642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":42642},{"type":"v1","addr":"172.21.15.196:6820","nonce":42642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":42642},{"type":"v1","addr":"172.21.15.196:6816","nonce":42642}]},"public_addr":"172.21.15.196:6809/42642","cluster_addr":"172.21.15.196:6812/42642","heartbeat_back_addr":"172.21.15.196:6820/42642","heartbeat_front_addr":"172.21.15.196:6816/42642","state":["exists","up"]},{"osd":2,"uuid":"2ede6fbd-035a-4829-bb99-a202a0992f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6811","nonce":42643},{"type":"v1","addr":"172.21.15.196:6813","nonce":42643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":42643},{"type":"v1","addr":"172.21.15.196:6817","nonce":42643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":42643},{"type":"v1","addr":"172.21.15.196:6823","nonce":42643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":42643},{"type":"v1","addr":"172.21.15.196:6821","nonce":42643}]},"public_addr":"172.21.15.196:6813/42643","cluster_addr":"172.21.15.196:6817/42643","heartbeat_back_addr":"172.21.15.196:6823/42643","heartbeat_front_addr":"172.21.15.196:6821/42643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:49:13.357 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-27T21:49:13.357 INFO:tasks.ceph:Creating RBD pool 2023-12-27T21:49:13.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-27T21:49:14.076 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'rbd' created 2023-12-27T21:49:14.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-27T21:49:15.083 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-27T21:49:15.097 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-27T21:49:15.098 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-27T21:49:15.098 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-27T21:49:15.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-27T21:49:15.099 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-27T21:49:15.437 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:49:15.450 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":45528},{"type":"v1","addr":"172.21.15.86:6827","nonce":45528}]},"active_addr":"172.21.15.86:6827/45528","active_change":"2023-12-27T21:49:09.251306+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3634193801}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2080208006}]}]} 2023-12-27T21:49:15.451 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-27T21:49:15.452 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-27T21:49:15.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-27T21:49:15.740 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:49:15.740 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":10,"fsid":"c088ab28-4839-43bd-96eb-95acf66420e1","created":"2023-12-27T21:49:05.091827+0000","modified":"2023-12-27T21:49:15.070119+0000","last_up_change":"2023-12-27T21:49:13.068458+0000","last_in_change":"2023-12-27T21:49:06.015055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-27T21:49:13.107754+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-27T21:49:13.678559+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7d01c453-d5a0-41a0-a464-d79d2985f9a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":42641},{"type":"v1","addr":"172.21.15.196:6801","nonce":42641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":42641},{"type":"v1","addr":"172.21.15.196:6803","nonce":42641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":42641},{"type":"v1","addr":"172.21.15.196:6807","nonce":42641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":42641},{"type":"v1","addr":"172.21.15.196:6805","nonce":42641}]},"public_addr":"172.21.15.196:6801/42641","cluster_addr":"172.21.15.196:6803/42641","heartbeat_back_addr":"172.21.15.196:6807/42641","heartbeat_front_addr":"172.21.15.196:6805/42641","state":["exists","up"]},{"osd":1,"uuid":"a16078f7-978e-40a5-8d09-00823ef94296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":42642},{"type":"v1","addr":"172.21.15.196:6809","nonce":42642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":42642},{"type":"v1","addr":"172.21.15.196:6812","nonce":42642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":42642},{"type":"v1","addr":"172.21.15.196:6820","nonce":42642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":42642},{"type":"v1","addr":"172.21.15.196:6816","nonce":42642}]},"public_addr":"172.21.15.196:6809/42642","cluster_addr":"172.21.15.196:6812/42642","heartbeat_back_addr":"172.21.15.196:6820/42642","heartbeat_front_addr":"172.21.15.196:6816/42642","state":["exists","up"]},{"osd":2,"uuid":"2ede6fbd-035a-4829-bb99-a202a0992f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6811","nonce":42643},{"type":"v1","addr":"172.21.15.196:6813","nonce":42643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":42643},{"type":"v1","addr":"172.21.15.196:6817","nonce":42643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":42643},{"type":"v1","addr":"172.21.15.196:6823","nonce":42643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":42643},{"type":"v1","addr":"172.21.15.196:6821","nonce":42643}]},"public_addr":"172.21.15.196:6813/42643","cluster_addr":"172.21.15.196:6817/42643","heartbeat_back_addr":"172.21.15.196:6823/42643","heartbeat_front_addr":"172.21.15.196:6821/42643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:49:11.130756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:49:11.174762+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:49:11.100290+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:49:15.753 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-27T21:49:15.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-27T21:49:16.040 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:49:16.041 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":10,"fsid":"c088ab28-4839-43bd-96eb-95acf66420e1","created":"2023-12-27T21:49:05.091827+0000","modified":"2023-12-27T21:49:15.070119+0000","last_up_change":"2023-12-27T21:49:13.068458+0000","last_in_change":"2023-12-27T21:49:06.015055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-27T21:49:13.107754+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-27T21:49:13.678559+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7d01c453-d5a0-41a0-a464-d79d2985f9a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":42641},{"type":"v1","addr":"172.21.15.196:6801","nonce":42641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":42641},{"type":"v1","addr":"172.21.15.196:6803","nonce":42641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":42641},{"type":"v1","addr":"172.21.15.196:6807","nonce":42641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":42641},{"type":"v1","addr":"172.21.15.196:6805","nonce":42641}]},"public_addr":"172.21.15.196:6801/42641","cluster_addr":"172.21.15.196:6803/42641","heartbeat_back_addr":"172.21.15.196:6807/42641","heartbeat_front_addr":"172.21.15.196:6805/42641","state":["exists","up"]},{"osd":1,"uuid":"a16078f7-978e-40a5-8d09-00823ef94296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":42642},{"type":"v1","addr":"172.21.15.196:6809","nonce":42642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":42642},{"type":"v1","addr":"172.21.15.196:6812","nonce":42642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":42642},{"type":"v1","addr":"172.21.15.196:6820","nonce":42642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":42642},{"type":"v1","addr":"172.21.15.196:6816","nonce":42642}]},"public_addr":"172.21.15.196:6809/42642","cluster_addr":"172.21.15.196:6812/42642","heartbeat_back_addr":"172.21.15.196:6820/42642","heartbeat_front_addr":"172.21.15.196:6816/42642","state":["exists","up"]},{"osd":2,"uuid":"2ede6fbd-035a-4829-bb99-a202a0992f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6811","nonce":42643},{"type":"v1","addr":"172.21.15.196:6813","nonce":42643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6815","nonce":42643},{"type":"v1","addr":"172.21.15.196:6817","nonce":42643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":42643},{"type":"v1","addr":"172.21.15.196:6823","nonce":42643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6819","nonce":42643},{"type":"v1","addr":"172.21.15.196:6821","nonce":42643}]},"public_addr":"172.21.15.196:6813/42643","cluster_addr":"172.21.15.196:6817/42643","heartbeat_back_addr":"172.21.15.196:6823/42643","heartbeat_front_addr":"172.21.15.196:6821/42643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:49:11.130756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:49:11.174762+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-27T21:49:11.100290+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-27T21:49:16.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-27T21:49:16.204 INFO:teuthology.orchestra.run.smithi196.stdout:34359738371 2023-12-27T21:49:16.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-27T21:49:16.353 INFO:teuthology.orchestra.run.smithi196.stdout:34359738370 2023-12-27T21:49:16.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-27T21:49:16.502 INFO:teuthology.orchestra.run.smithi196.stdout:34359738370 2023-12-27T21:49:16.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-27T21:49:16.789 INFO:teuthology.orchestra.run.smithi196.stdout:34359738370 2023-12-27T21:49:16.803 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2023-12-27T21:49:17.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-27T21:49:18.107 INFO:teuthology.orchestra.run.smithi196.stdout:34359738370 2023-12-27T21:49:18.121 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2023-12-27T21:49:19.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-27T21:49:19.422 INFO:teuthology.orchestra.run.smithi196.stdout:34359738372 2023-12-27T21:49:19.436 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738372 for osd.0 2023-12-27T21:49:19.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-27T21:49:19.725 INFO:teuthology.orchestra.run.smithi196.stdout:34359738371 2023-12-27T21:49:19.738 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738370 got 34359738371 for osd.1 2023-12-27T21:49:19.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-27T21:49:20.027 INFO:teuthology.orchestra.run.smithi196.stdout:34359738371 2023-12-27T21:49:20.041 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738370 got 34359738371 for osd.2 2023-12-27T21:49:20.041 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-27T21:49:20.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-27T21:49:20.329 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:49:20.330 INFO:teuthology.orchestra.run.smithi196.stderr:dumped all 2023-12-27T21:49:20.342 INFO:teuthology.orchestra.run.smithi196.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-12-27T21:49:19.256575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223944,"statfs":{"total":289910292480,"available":288997318656,"internally_reserved":0,"allocated":294912,"data_stored":80424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.091300+0000","last_change":"2023-12-27T21:49:15.085196+0000","last_active":"2023-12-27T21:49:15.091300+0000","last_peered":"2023-12-27T21:49:15.091300+0000","last_clean":"2023-12-27T21:49:15.091300+0000","last_became_active":"2023-12-27T21:49:15.084906+0000","last_became_peered":"2023-12-27T21:49:15.084906+0000","last_unstale":"2023-12-27T21:49:15.091300+0000","last_undegraded":"2023-12-27T21:49:15.091300+0000","last_fullsized":"2023-12-27T21:49:15.091300+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.092944+0000","last_change":"2023-12-27T21:49:15.090830+0000","last_active":"2023-12-27T21:49:15.092944+0000","last_peered":"2023-12-27T21:49:15.092944+0000","last_clean":"2023-12-27T21:49:15.092944+0000","last_became_active":"2023-12-27T21:49:15.090516+0000","last_became_peered":"2023-12-27T21:49:15.090516+0000","last_unstale":"2023-12-27T21:49:15.092944+0000","last_undegraded":"2023-12-27T21:49:15.092944+0000","last_fullsized":"2023-12-27T21:49:15.092944+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.092947+0000","last_change":"2023-12-27T21:49:15.090824+0000","last_active":"2023-12-27T21:49:15.092947+0000","last_peered":"2023-12-27T21:49:15.092947+0000","last_clean":"2023-12-27T21:49:15.092947+0000","last_became_active":"2023-12-27T21:49:15.090420+0000","last_became_peered":"2023-12-27T21:49:15.090420+0000","last_unstale":"2023-12-27T21:49:15.092947+0000","last_undegraded":"2023-12-27T21:49:15.092947+0000","last_fullsized":"2023-12-27T21:49:15.092947+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.089834+0000","last_change":"2023-12-27T21:49:15.082844+0000","last_active":"2023-12-27T21:49:15.089834+0000","last_peered":"2023-12-27T21:49:15.089834+0000","last_clean":"2023-12-27T21:49:15.089834+0000","last_became_active":"2023-12-27T21:49:15.082645+0000","last_became_peered":"2023-12-27T21:49:15.082645+0000","last_unstale":"2023-12-27T21:49:15.089834+0000","last_undegraded":"2023-12-27T21:49:15.089834+0000","last_fullsized":"2023-12-27T21:49:15.089834+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.089872+0000","last_change":"2023-12-27T21:49:15.083902+0000","last_active":"2023-12-27T21:49:15.089872+0000","last_peered":"2023-12-27T21:49:15.089872+0000","last_clean":"2023-12-27T21:49:15.089872+0000","last_became_active":"2023-12-27T21:49:15.083355+0000","last_became_peered":"2023-12-27T21:49:15.083355+0000","last_unstale":"2023-12-27T21:49:15.089872+0000","last_undegraded":"2023-12-27T21:49:15.089872+0000","last_fullsized":"2023-12-27T21:49:15.089872+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.091259+0000","last_change":"2023-12-27T21:49:15.085346+0000","last_active":"2023-12-27T21:49:15.091259+0000","last_peered":"2023-12-27T21:49:15.091259+0000","last_clean":"2023-12-27T21:49:15.091259+0000","last_became_active":"2023-12-27T21:49:15.084944+0000","last_became_peered":"2023-12-27T21:49:15.084944+0000","last_unstale":"2023-12-27T21:49:15.091259+0000","last_undegraded":"2023-12-27T21:49:15.091259+0000","last_fullsized":"2023-12-27T21:49:15.091259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.089914+0000","last_change":"2023-12-27T21:49:15.083578+0000","last_active":"2023-12-27T21:49:15.089914+0000","last_peered":"2023-12-27T21:49:15.089914+0000","last_clean":"2023-12-27T21:49:15.089914+0000","last_became_active":"2023-12-27T21:49:15.083264+0000","last_became_peered":"2023-12-27T21:49:15.083264+0000","last_unstale":"2023-12-27T21:49:15.089914+0000","last_undegraded":"2023-12-27T21:49:15.089914+0000","last_fullsized":"2023-12-27T21:49:15.089914+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.090447+0000","last_change":"2023-12-27T21:49:15.085518+0000","last_active":"2023-12-27T21:49:15.090447+0000","last_peered":"2023-12-27T21:49:15.090447+0000","last_clean":"2023-12-27T21:49:15.090447+0000","last_became_active":"2023-12-27T21:49:15.085236+0000","last_became_peered":"2023-12-27T21:49:15.085236+0000","last_unstale":"2023-12-27T21:49:15.090447+0000","last_undegraded":"2023-12-27T21:49:15.090447+0000","last_fullsized":"2023-12-27T21:49:15.090447+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.092936+0000","last_change":"2023-12-27T21:49:15.090776+0000","last_active":"2023-12-27T21:49:15.092936+0000","last_peered":"2023-12-27T21:49:15.092936+0000","last_clean":"2023-12-27T21:49:15.092936+0000","last_became_active":"2023-12-27T21:49:15.090451+0000","last_became_peered":"2023-12-27T21:49:15.090451+0000","last_unstale":"2023-12-27T21:49:15.092936+0000","last_undegraded":"2023-12-27T21:49:15.092936+0000","last_fullsized":"2023-12-27T21:49:15.092936+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-27T21:49:20.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-27T21:49:20.627 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:49:20.628 INFO:teuthology.orchestra.run.smithi196.stderr:dumped all 2023-12-27T21:49:20.639 INFO:teuthology.orchestra.run.smithi196.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-12-27T21:49:19.256575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223944,"statfs":{"total":289910292480,"available":288997318656,"internally_reserved":0,"allocated":294912,"data_stored":80424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.091300+0000","last_change":"2023-12-27T21:49:15.085196+0000","last_active":"2023-12-27T21:49:15.091300+0000","last_peered":"2023-12-27T21:49:15.091300+0000","last_clean":"2023-12-27T21:49:15.091300+0000","last_became_active":"2023-12-27T21:49:15.084906+0000","last_became_peered":"2023-12-27T21:49:15.084906+0000","last_unstale":"2023-12-27T21:49:15.091300+0000","last_undegraded":"2023-12-27T21:49:15.091300+0000","last_fullsized":"2023-12-27T21:49:15.091300+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.092944+0000","last_change":"2023-12-27T21:49:15.090830+0000","last_active":"2023-12-27T21:49:15.092944+0000","last_peered":"2023-12-27T21:49:15.092944+0000","last_clean":"2023-12-27T21:49:15.092944+0000","last_became_active":"2023-12-27T21:49:15.090516+0000","last_became_peered":"2023-12-27T21:49:15.090516+0000","last_unstale":"2023-12-27T21:49:15.092944+0000","last_undegraded":"2023-12-27T21:49:15.092944+0000","last_fullsized":"2023-12-27T21:49:15.092944+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.092947+0000","last_change":"2023-12-27T21:49:15.090824+0000","last_active":"2023-12-27T21:49:15.092947+0000","last_peered":"2023-12-27T21:49:15.092947+0000","last_clean":"2023-12-27T21:49:15.092947+0000","last_became_active":"2023-12-27T21:49:15.090420+0000","last_became_peered":"2023-12-27T21:49:15.090420+0000","last_unstale":"2023-12-27T21:49:15.092947+0000","last_undegraded":"2023-12-27T21:49:15.092947+0000","last_fullsized":"2023-12-27T21:49:15.092947+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.089834+0000","last_change":"2023-12-27T21:49:15.082844+0000","last_active":"2023-12-27T21:49:15.089834+0000","last_peered":"2023-12-27T21:49:15.089834+0000","last_clean":"2023-12-27T21:49:15.089834+0000","last_became_active":"2023-12-27T21:49:15.082645+0000","last_became_peered":"2023-12-27T21:49:15.082645+0000","last_unstale":"2023-12-27T21:49:15.089834+0000","last_undegraded":"2023-12-27T21:49:15.089834+0000","last_fullsized":"2023-12-27T21:49:15.089834+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.089872+0000","last_change":"2023-12-27T21:49:15.083902+0000","last_active":"2023-12-27T21:49:15.089872+0000","last_peered":"2023-12-27T21:49:15.089872+0000","last_clean":"2023-12-27T21:49:15.089872+0000","last_became_active":"2023-12-27T21:49:15.083355+0000","last_became_peered":"2023-12-27T21:49:15.083355+0000","last_unstale":"2023-12-27T21:49:15.089872+0000","last_undegraded":"2023-12-27T21:49:15.089872+0000","last_fullsized":"2023-12-27T21:49:15.089872+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.091259+0000","last_change":"2023-12-27T21:49:15.085346+0000","last_active":"2023-12-27T21:49:15.091259+0000","last_peered":"2023-12-27T21:49:15.091259+0000","last_clean":"2023-12-27T21:49:15.091259+0000","last_became_active":"2023-12-27T21:49:15.084944+0000","last_became_peered":"2023-12-27T21:49:15.084944+0000","last_unstale":"2023-12-27T21:49:15.091259+0000","last_undegraded":"2023-12-27T21:49:15.091259+0000","last_fullsized":"2023-12-27T21:49:15.091259+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.089914+0000","last_change":"2023-12-27T21:49:15.083578+0000","last_active":"2023-12-27T21:49:15.089914+0000","last_peered":"2023-12-27T21:49:15.089914+0000","last_clean":"2023-12-27T21:49:15.089914+0000","last_became_active":"2023-12-27T21:49:15.083264+0000","last_became_peered":"2023-12-27T21:49:15.083264+0000","last_unstale":"2023-12-27T21:49:15.089914+0000","last_undegraded":"2023-12-27T21:49:15.089914+0000","last_fullsized":"2023-12-27T21:49:15.089914+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.090447+0000","last_change":"2023-12-27T21:49:15.085518+0000","last_active":"2023-12-27T21:49:15.090447+0000","last_peered":"2023-12-27T21:49:15.090447+0000","last_clean":"2023-12-27T21:49:15.090447+0000","last_became_active":"2023-12-27T21:49:15.085236+0000","last_became_peered":"2023-12-27T21:49:15.085236+0000","last_unstale":"2023-12-27T21:49:15.090447+0000","last_undegraded":"2023-12-27T21:49:15.090447+0000","last_fullsized":"2023-12-27T21:49:15.090447+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-27T21:49:15.092936+0000","last_change":"2023-12-27T21:49:15.090776+0000","last_active":"2023-12-27T21:49:15.092936+0000","last_peered":"2023-12-27T21:49:15.092936+0000","last_clean":"2023-12-27T21:49:15.092936+0000","last_became_active":"2023-12-27T21:49:15.090451+0000","last_became_peered":"2023-12-27T21:49:15.090451+0000","last_unstale":"2023-12-27T21:49:15.092936+0000","last_undegraded":"2023-12-27T21:49:15.092936+0000","last_fullsized":"2023-12-27T21:49:15.092936+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-27T21:49:14.069181+0000","last_clean_scrub_stamp":"2023-12-27T21:49:14.069181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-27T21:49:20.640 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-27T21:49:20.640 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-27T21:49:20.640 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-27T21:49:20.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-27T21:49:20.965 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-12-27T21:49:20.979 INFO:teuthology.orchestra.run.smithi196.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-27T21:49:20.979 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-27T21:49:20.979 INFO:teuthology.run_tasks:Running task exec... 2023-12-27T21:49:20.990 INFO:teuthology.task.exec:Executing custom commands... 2023-12-27T21:49:20.990 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi196.front.sepia.ceph.com 2023-12-27T21:49:20.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:21.358 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror 2023-12-27T21:49:21.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:21.737 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.0 2023-12-27T21:49:21.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:22.110 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.1 2023-12-27T21:49:22.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:22.488 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.2 2023-12-27T21:49:22.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:22.864 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.3 2023-12-27T21:49:22.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:23.246 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror 2023-12-27T21:49:23.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:23.626 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.0 2023-12-27T21:49:23.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:24.002 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.1 2023-12-27T21:49:24.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:24.383 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.2 2023-12-27T21:49:24.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-27T21:49:24.759 INFO:teuthology.orchestra.run.smithi196.stderr:updated caps for client.mirror.3 2023-12-27T21:49:24.770 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T21:49:24.782 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T21:49:24.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T21:49:24.784 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-27T21:49:24.800 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-27T21:49:24.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-27T21:49:24.802 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-27T21:49:24.810 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-27T21:49:24.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-27T21:49:24.812 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-27T21:49:24.820 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-27T21:49:24.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-27T21:49:24.823 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T21:49:24.832 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T21:49:24.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T21:49:24.835 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-27T21:49:24.848 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-27T21:49:24.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-27T21:49:24.850 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-27T21:49:24.860 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-27T21:49:24.867 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-27T21:49:24.867 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-27T21:49:24.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-27T21:49:24.876 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-27T21:49:24.879 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-27T21:49:24.880 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-27T21:49:24.891 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1703713764 2023-12-27T21:49:24.891 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-27T21:49:24.891 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-27T21:49:24.900 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1703713764 2023-12-27T21:49:24.900 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-27T21:49:24.901 INFO:teuthology.run_tasks:Running task workunit... 2023-12-27T21:49:24.912 INFO:tasks.workunit:Pulling workunits from ref c5564c7988cbaadc3382253af5843a8595347c2d 2023-12-27T21:49:24.913 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-27T21:49:24.913 DEBUG:teuthology.orchestra.run.smithi196:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-27T21:49:24.913 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-27T21:49:24.914 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-27T21:49:24.914 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-27T21:49:24.914 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-27T21:49:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:49:24.936 INFO:teuthology.orchestra.run.smithi196.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-27T21:49:24.936 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-27T21:49:24.992 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-27T21:49:24.993 DEBUG:teuthology.orchestra.run.smithi196:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-27T21:49:25.049 INFO:tasks.workunit:timeout=6h 2023-12-27T21:49:25.050 INFO:tasks.workunit:cleanup=True 2023-12-27T21:49:25.050 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout c5564c7988cbaadc3382253af5843a8595347c2d 2023-12-27T21:49:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-27T21:50:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Note: switching to 'c5564c7988cbaadc3382253af5843a8595347c2d'. 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:state without impacting any branches by switching back to a branch. 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: git switch -c 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Or undo this operation with: 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:50:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: git switch - 2023-12-27T21:50:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:50:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-27T21:50:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:50:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:HEAD is now at c5564c7988c Merge branch 'wip-53897-pacific' of https://github.com/idryomov/ceph into wip-yuri7-testing-2023-12-27-1008-pacific 2023-12-27T21:50:06.670 DEBUG:teuthology.orchestra.run.smithi196:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-27T21:50:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-27T21:50:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-27T21:50:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-27T21:50:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-27T21:50:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-27T21:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-27T21:50:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-27T21:50:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-27T21:50:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-27T21:50:07.329 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-12-27T21:50:07.329 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-27T21:50:07.387 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-27T21:50:07.388 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-27T21:50:07.389 DEBUG:teuthology.orchestra.run.smithi196:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5564c7988cbaadc3382253af5843a8595347c2d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-27T21:50:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ IMAGE_COUNT=50 2023-12-27T21:50:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ export LOCKDEP=0 2023-12-27T21:50:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ LOCKDEP=0 2023-12-27T21:50:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ type xmlstarlet 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ XMLSTARLET=xmlstarlet 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ RBD_MIRROR_INSTANCES=4 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CLUSTER1=cluster1 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CLUSTER2=cluster2 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ PEER_CLUSTER_SUFFIX= 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ POOL=mirror 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ PARENT_POOL=mirror_parent 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ NS1=ns1 2023-12-27T21:50:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ NS2=ns2 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ TEMPDIR= 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ID=mirror 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ MIRROR_POOL_MODE=image 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ MIRROR_IMAGE_MODE=snapshot 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS='--id mirror' 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ LAST_MIRROR_INSTANCE=3 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-27T21:50:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-27T21:50:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-27T21:50:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_BIN=. 2023-12-27T21:50:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_SRC=. 2023-12-27T21:50:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' -e CMakeCache.txt ']' 2023-12-27T21:50:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 -gt 0 ']' 2023-12-27T21:50:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ setup 2023-12-27T21:50:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local c 2023-12-27T21:50:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-12-27T21:50:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ setup_tempdir 2023-12-27T21:50:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mktemp -d 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ TEMPDIR=/tmp/tmp.yuhsqMaECB 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -z 1 ']' 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ setup_pools cluster1 cluster2 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local remote_cluster=cluster2 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mon_map_file 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mon_addr 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local admin_key_file 2023-12-27T21:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local uuid 2023-12-27T21:50:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-27T21:50:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:pool 'mirror' already exists 2023-12-27T21:50:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-27T21:50:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:pool 'mirror_parent' already exists 2023-12-27T21:50:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 pool init mirror 2023-12-27T21:50:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-12-27T21:50:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-27T21:50:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-27T21:50:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-27T21:50:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-27T21:50:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-27T21:50:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -z ']' 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -z ']' 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_add cluster1 mirror cluster2 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local client_cluster=cluster2 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local remote_cluster=cluster2 2023-12-27T21:50:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local uuid_var_name 2023-12-27T21:50:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local error_code 2023-12-27T21:50:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local peer_uuid 2023-12-27T21:50:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 16 32 2023-12-27T21:50:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:50:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_uuid=d61dbc68-cd84-40fa-bb5c-da472dfc1d1b 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=0 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 0 -eq 17 ']' 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 0 -eq 0 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror_parent 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local client_cluster=cluster2 2023-12-27T21:50:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local remote_cluster=cluster2 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local uuid_var_name 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local error_code 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local peer_uuid 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 16 32 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:50:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_uuid=b180f729-747f-461c-b0f6-1837ed333fe4 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=0 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 0 -eq 17 ']' 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 0 -eq 0 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ setup_pools cluster2 cluster1 2023-12-27T21:50:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local remote_cluster=cluster1 2023-12-27T21:50:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mon_map_file 2023-12-27T21:50:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mon_addr 2023-12-27T21:50:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local admin_key_file 2023-12-27T21:50:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local uuid 2023-12-27T21:50:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-27T21:50:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:pool 'mirror' already exists 2023-12-27T21:50:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-27T21:50:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:pool 'mirror_parent' already exists 2023-12-27T21:50:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 pool init mirror 2023-12-27T21:50:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-12-27T21:50:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-27T21:50:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-27T21:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-27T21:50:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-27T21:50:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-27T21:50:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-27T21:50:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -z ']' 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -z ']' 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_add cluster2 mirror cluster1 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local client_cluster=cluster1 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local remote_cluster=cluster1 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local uuid_var_name 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local error_code 2023-12-27T21:50:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local peer_uuid 2023-12-27T21:50:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 16 32 2023-12-27T21:50:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:50:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-27T21:50:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_uuid=d5437fd2-985a-4ea7-bfca-246d2068d591 2023-12-27T21:50:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=0 2023-12-27T21:50:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 0 -eq 17 ']' 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 0 -eq 0 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror_parent 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local client_cluster=cluster1 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local remote_cluster=cluster1 2023-12-27T21:50:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local uuid_var_name 2023-12-27T21:50:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local error_code 2023-12-27T21:50:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local peer_uuid 2023-12-27T21:50:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 16 32 2023-12-27T21:50:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:50:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ peer_uuid=aec83290-50e1-44dd-bd9f-7eb761111b0d 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=0 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 0 -eq 17 ']' 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 0 -eq 0 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirrors cluster1 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 0 3 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster1:0 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1:0 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster1:0 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster1 2023-12-27T21:50:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=0 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 0 = cluster1:0 ']' 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster1 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=0 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster1:1 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1:1 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster1:1 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster1 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=1 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 1 = cluster1:1 ']' 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster1 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=1 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster1:2 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1:2 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:50:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster1:2 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster1 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=2 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 2 = cluster1:2 ']' 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster1 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=2 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster1:3 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1:3 2023-12-27T21:50:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster1:3 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster1 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=3 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 3 = cluster1:3 ']' 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster1 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=3 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirrors cluster2 2023-12-27T21:50:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 0 3 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster2:0 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2:0 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster2:0 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster2 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=0 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 0 = cluster2:0 ']' 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster2 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster2 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=0 2023-12-27T21:50:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster2:1 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2:1 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster2:1 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster2 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=1 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 1 = cluster2:1 ']' 2023-12-27T21:50:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster2 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster2 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=1 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster2:2 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2:2 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster2:2 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster2 2023-12-27T21:50:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=2 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 2 = cluster2:2 ']' 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster2 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster2 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=2 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster2:3 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2:3 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster2:3 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T21:50:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster2 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=3 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 3 = cluster2:3 ']' 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster2 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster2 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=3 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee -a /tmp/tmp.yuhsqMaECB/rbd-mirror.test.log 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ date '+%F %T' 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo 2023-12-27 21:50:23 TEST: add image and test replay after client crashes 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-27 21:50:23 TEST: add image and test replay after client crashes 2023-12-27T21:50:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=test 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror test 512M 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T21:50:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T21:50:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T21:50:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T21:50:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2023-12-27T21:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T21:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-27T21:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T21:50:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-27T21:50:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T21:50:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-27T21:50:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:50:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:50:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:50:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1099ebd91038 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1099ebd91038 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:23 2023 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:50:23 2023 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:23 2023 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:50:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T21:50:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:50:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:50:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:50:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:50:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:50:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:50:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:50:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:50:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:50:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:50:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:50:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:50:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:50:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:50:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:50:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:50:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:50:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:50:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:50:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:50:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:50:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:50:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:50:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:50:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:50:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:50:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:50:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:50:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:50:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:50:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:50:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:50:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:50:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:50:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:50:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:50:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:50:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 1 2023-12-27T21:50:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:50:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:50:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:50:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:50:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:50:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:50:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:50:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:50:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:50:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:50:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:50:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:50:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:50:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:50:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:50:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:50:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:50:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:50:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:50:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:50:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:50:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:50:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:50:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:50:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:50:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:50:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:50:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:50:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:50:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:50:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:50:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:50:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:50:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 2 2023-12-27T21:50:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:50:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:50:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:50:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:50:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:50:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:50:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:50:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:50:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:50:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:50:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:50:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:50:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:50:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:50:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name= 2023-12-27T21:50:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 10 2023-12-27T21:50:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:50:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:50:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:50:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T21:50:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:50:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:50:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 22441984 bytes 2023-12-27T21:50:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 103610368 bytes 2023-12-27T21:50:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 189198336 bytes 2023-12-27T21:50:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 239279104 bytes 2023-12-27T21:50:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 289267712 bytes 2023-12-27T21:50:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 334934016 bytes 2023-12-27T21:50:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 398696448 bytes 2023-12-27T21:50:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 433046528 bytes 2023-12-27T21:50:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 484775936 bytes 2023-12-27T21:50:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 593538048 bytes 2023-12-27T21:50:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 663781376 bytes 2023-12-27T21:50:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 731904000 bytes 2023-12-27T21:50:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 744819712 bytes 2023-12-27T21:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 775891968 bytes 2023-12-27T21:50:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 806748160 bytes 2023-12-27T21:50:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 823139328 bytes 2023-12-27T21:50:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 866846720 bytes 2023-12-27T21:50:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 919593984 bytes 2023-12-27T21:50:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 981977088 bytes 2023-12-27T21:50:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1053288448 bytes 2023-12-27T21:50:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1069477888 bytes 2023-12-27T21:50:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1131113472 bytes 2023-12-27T21:50:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1174769664 bytes 2023-12-27T21:50:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1196138496 bytes 2023-12-27T21:50:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1277571072 bytes 2023-12-27T21:50:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1290861568 bytes 2023-12-27T21:50:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1342646272 bytes 2023-12-27T21:50:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1397368832 bytes 2023-12-27T21:50:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1502229504 bytes 2023-12-27T21:50:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1578936320 bytes 2023-12-27T21:50:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1646915584 bytes 2023-12-27T21:50:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1751648256 bytes 2023-12-27T21:50:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1843149824 bytes 2023-12-27T21:50:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1853340672 bytes 2023-12-27T21:50:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1894754304 bytes 2023-12-27T21:50:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1903311872 bytes 2023-12-27T21:50:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1944439808 bytes 2023-12-27T21:50:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1972294656 bytes 2023-12-27T21:50:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1991764992 bytes 2023-12-27T21:50:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 2020715520 bytes 2023-12-27T21:50:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 2058779648 bytes 2023-12-27T21:50:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 2077803520 bytes 2023-12-27T21:50:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 2119780352 bytes 2023-12-27T21:50:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2137356288 bytes 2023-12-27T21:50:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2204177408 bytes 2023-12-27T21:50:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2223817728 bytes 2023-12-27T21:50:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2224424960 bytes 2023-12-27T21:50:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2225032192 bytes 2023-12-27T21:50:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2323881984 bytes 2023-12-27T21:50:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2370306048 bytes 2023-12-27T21:50:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2415896576 bytes 2023-12-27T21:50:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2447771648 bytes 2023-12-27T21:50:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2489148416 bytes 2023-12-27T21:50:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2531723264 bytes 2023-12-27T21:50:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2563707904 bytes 2023-12-27T21:50:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2591617024 bytes 2023-12-27T21:50:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2645292032 bytes 2023-12-27T21:50:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2688708608 bytes 2023-12-27T21:50:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2757544960 bytes 2023-12-27T21:50:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2794961920 bytes 2023-12-27T21:50:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2891425792 bytes 2023-12-27T21:50:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2966834176 bytes 2023-12-27T21:50:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 3025456128 bytes 2023-12-27T21:50:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 3077957632 bytes 2023-12-27T21:50:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 3108639744 bytes 2023-12-27T21:50:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 3130425344 bytes 2023-12-27T21:50:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 3147642880 bytes 2023-12-27T21:50:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 3149646848 bytes 2023-12-27T21:50:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 3198876672 bytes 2023-12-27T21:50:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 3238658048 bytes 2023-12-27T21:50:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3266244608 bytes 2023-12-27T21:50:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3279621120 bytes 2023-12-27T21:50:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3326943232 bytes 2023-12-27T21:50:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3329435648 bytes 2023-12-27T21:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3355706368 bytes 2023-12-27T21:50:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3396968448 bytes 2023-12-27T21:50:41.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3406521344 bytes 2023-12-27T21:50:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3448529920 bytes 2023-12-27T21:50:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3498402816 bytes 2023-12-27T21:50:41.915 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T21:50:41.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T21:50:41.916 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-27T21:50:41.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T21:50:41.916+0000 7f82a1b98700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 46430) UID: 1000 2023-12-27T21:50:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3525760000 bytes 2023-12-27T21:50:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8100 IOs, 3535683584 bytes 2023-12-27T21:50:42.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:50:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8200 IOs, 3586981888 bytes 2023-12-27T21:50:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8300 IOs, 3605093376 bytes 2023-12-27T21:50:42.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8400 IOs, 3645256704 bytes 2023-12-27T21:50:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8500 IOs, 3665210368 bytes 2023-12-27T21:50:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8600 IOs, 3740791808 bytes 2023-12-27T21:50:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8700 IOs, 3775194112 bytes 2023-12-27T21:50:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8800 IOs, 3817612288 bytes 2023-12-27T21:50:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:50:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:50:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:50:43.917 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T21:50:43.917 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:50:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:50:43.918 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T21:50:43.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T21:50:43.918 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T21:50:43.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T21:50:43.922 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-12-27T21:50:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:50:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:50:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:50:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:50:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:50:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:50:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: down+unknown 2023-12-27T21:50:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: status not found 2023-12-27T21:50:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27T21:50:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:50:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:50:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: down+unknown 2023-12-27T21:50:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: status not found 2023-12-27T21:50:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27T21:50:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 1 2023-12-27T21:50:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:50:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 2 2023-12-27T21:50:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:50:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:50:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:50:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:50:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:50:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:50:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:50:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1703713824,"remote_snapshot_timestamp":1703713824,"replay_state":"idle"} 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.3 on smithi196 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:50:46 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: down+unknown 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: status not found 2023-12-27T21:50:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27T21:50:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:50:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1703713824,"remote_snapshot_timestamp":1703713824,"replay_state":"idle"} 2023-12-27T21:50:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:50:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:50:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:50:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:50:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.3 on smithi196 2023-12-27T21:50:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:50:46' 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:50:46' 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:50:46' 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:50:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:50:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:50:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:50:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:50:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:50:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:50:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:50:46' 2023-12-27T21:50:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:50:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:50:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:50:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:50:46.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:50:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:50:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap1 2023-12-27T21:50:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap1 2023-12-27T21:50:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:50:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:50:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:50:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap1 2023-12-27T21:50:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:51:04.915 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-27T21:51:05.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-27T21:51:05.022 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-27T21:51:05.023 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-27T21:51:05.023 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:51:05.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T21:51:05.025+0000 7f1bacb96700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 46508) UID: 1000 2023-12-27T21:51:05.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:51:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:51:11.125 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-27T21:51:11.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T21:51:11.126 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T21:51:11.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T21:51:11.129 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-27T21:51:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:51:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:51:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:51:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:51:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:51:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:51:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:51:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:51:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:51:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:51:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:51:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:51:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:51:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:51:13.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:51:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:51:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:51:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:51:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:51:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:51:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:51:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:51:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:51:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:51:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:51:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:51:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:51:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:51:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:51:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:51:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:51:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:51:13.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:51:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:51:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:51:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:51:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:51:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:51:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:51:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:51:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:51:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:51:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:51:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:51:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:51:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:51:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:51:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:51:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:51:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:51:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:51:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:51:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:51:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:51:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:51:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:51:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 6 2023-12-27T21:51:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:51:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:51:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:51:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:51:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:51:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=6 2023-12-27T21:51:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:51:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:51:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:51:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:51:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:51:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:51:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=4 2023-12-27T21:51:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 6 = 4 2023-12-27T21:51:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:51:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:51:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:51:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:51:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:51:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=4 2023-12-27T21:51:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 6 = 4 2023-12-27T21:51:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:51:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:51:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:51:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:51:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:51:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:51:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:51:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:51:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:51:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:51:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 6 = '' 2023-12-27T21:51:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:51:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:51:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:51:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:51:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=6 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 6 = 6 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap1 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:51:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:51:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-27T21:51:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:51:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:51:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:51:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap1 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap1 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:51:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:51:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:51:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:51:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:51:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:51:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:51:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:51:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:51:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap1-clean 2023-12-27T21:51:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-27T21:51:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap1-clean 2023-12-27T21:51:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:51:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:51:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:51:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:51:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T21:51:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:51:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:51:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 73477120 bytes 2023-12-27T21:51:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 117788672 bytes 2023-12-27T21:51:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 165810176 bytes 2023-12-27T21:51:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 214525952 bytes 2023-12-27T21:51:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 247003136 bytes 2023-12-27T21:51:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 279822336 bytes 2023-12-27T21:51:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 362156032 bytes 2023-12-27T21:51:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 430087168 bytes 2023-12-27T21:51:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 451105792 bytes 2023-12-27T21:51:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 455557120 bytes 2023-12-27T21:51:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 479939584 bytes 2023-12-27T21:51:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 483505152 bytes 2023-12-27T21:51:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 523572224 bytes 2023-12-27T21:51:23.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 553338880 bytes 2023-12-27T21:51:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 580224000 bytes 2023-12-27T21:51:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 618554368 bytes 2023-12-27T21:51:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 668003328 bytes 2023-12-27T21:51:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 668609536 bytes 2023-12-27T21:51:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 724365312 bytes 2023-12-27T21:51:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 768955392 bytes 2023-12-27T21:51:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 794412032 bytes 2023-12-27T21:51:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 836549632 bytes 2023-12-27T21:51:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 868001792 bytes 2023-12-27T21:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 903937024 bytes 2023-12-27T21:51:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 933655552 bytes 2023-12-27T21:51:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 960542720 bytes 2023-12-27T21:51:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1006248960 bytes 2023-12-27T21:51:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1023709184 bytes 2023-12-27T21:51:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1093568512 bytes 2023-12-27T21:51:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1108809728 bytes 2023-12-27T21:51:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1142889472 bytes 2023-12-27T21:51:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1180094464 bytes 2023-12-27T21:51:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1220791296 bytes 2023-12-27T21:51:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1239689216 bytes 2023-12-27T21:51:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1294631936 bytes 2023-12-27T21:51:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1355735040 bytes 2023-12-27T21:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1356356608 bytes 2023-12-27T21:51:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1440618496 bytes 2023-12-27T21:51:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1493149696 bytes 2023-12-27T21:51:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1494554624 bytes 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:51:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:51:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:51:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:51:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:51:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:51:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:51:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:51:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:51:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:51:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:51:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:51:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":17895048.53,"bytes_per_snapshot":536851456.0,"last_snapshot_bytes":536851456,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703713874,"remote_snapshot_timestamp":1703713874,"replay_state":"idle"} 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.3 on smithi196 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:51:24 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:51:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:51:25 2023-12-27T21:51:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:51:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":17895048.53,"bytes_per_snapshot":536851456.0,"last_snapshot_bytes":536851456,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703713874,"remote_snapshot_timestamp":1703713874,"replay_state":"idle"} 2023-12-27T21:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:51:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:51:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.3 on smithi196 2023-12-27T21:51:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:51:24' 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:51:24' 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:51:24' 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:51:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:51:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:51:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:51:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:51:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:51:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:51:24' 2023-12-27T21:51:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:51:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:51:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:51:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:51:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:51:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap2 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap2 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap2 2023-12-27T21:51:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:51:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:51:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:51:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:51:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:51:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:51:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:51:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:51:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:51:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:51:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:51:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:51:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:51:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:51:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:51:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:51:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:51:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:51:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:51:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:51:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:51:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:51:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:51:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:51:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:51:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:51:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:51:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:51:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:51:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:51:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:51:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:51:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:51:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:51:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:51:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:51:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:51:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:51:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:51:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:51:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:51:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:51:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:51:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:51:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:51:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:51:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:51:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:51:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:51:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:51:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:51:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:51:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:51:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:51:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:51:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:51:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:51:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:51:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:51:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:51:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:51:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:51:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:51:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:51:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:52:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 9 2023-12-27T21:52:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:52:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:52:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:52:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:52:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=9 2023-12-27T21:52:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:52:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:52:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:52:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:52:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:52:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:52:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 9 = '' 2023-12-27T21:52:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:52:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:52:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:52:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:52:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:52:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:52:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 9 = '' 2023-12-27T21:52:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:52:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:52:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:52:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:52:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:52:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:52:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 9 = '' 2023-12-27T21:52:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:52:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:52:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:52:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:52:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:52:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:52:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 9 = '' 2023-12-27T21:52:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 2 2023-12-27T21:52:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:52:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=9 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 9 = 9 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap2 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:52:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:52:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-27T21:52:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:52:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 4 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:52:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:52:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:52:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap1-clean ']' 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap1-clean 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:52:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:52:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:52:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap2 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap2 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:52:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:52:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:52:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:52:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap2-clean 2023-12-27T21:52:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-27T21:52:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap2-clean 2023-12-27T21:52:10.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:52:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:52:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:52:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:11.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:52:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T21:52:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:52:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:52:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 72850432 bytes 2023-12-27T21:52:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 152232960 bytes 2023-12-27T21:52:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 259817472 bytes 2023-12-27T21:52:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 308275200 bytes 2023-12-27T21:52:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 334914560 bytes 2023-12-27T21:52:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 335509504 bytes 2023-12-27T21:52:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 378427392 bytes 2023-12-27T21:52:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 415292416 bytes 2023-12-27T21:52:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 456499200 bytes 2023-12-27T21:52:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 523416576 bytes 2023-12-27T21:52:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 631865344 bytes 2023-12-27T21:52:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 665527296 bytes 2023-12-27T21:52:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 699429888 bytes 2023-12-27T21:52:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 744495104 bytes 2023-12-27T21:52:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 806438912 bytes 2023-12-27T21:52:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 883832832 bytes 2023-12-27T21:52:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 950921216 bytes 2023-12-27T21:52:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 989390848 bytes 2023-12-27T21:52:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 1032740864 bytes 2023-12-27T21:52:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1112367104 bytes 2023-12-27T21:52:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1157019648 bytes 2023-12-27T21:52:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1250936832 bytes 2023-12-27T21:52:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1338120192 bytes 2023-12-27T21:52:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1373996032 bytes 2023-12-27T21:52:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1459075072 bytes 2023-12-27T21:52:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1514454016 bytes 2023-12-27T21:52:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1522167808 bytes 2023-12-27T21:52:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1608070144 bytes 2023-12-27T21:52:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1651005440 bytes 2023-12-27T21:52:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1672059904 bytes 2023-12-27T21:52:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1727277056 bytes 2023-12-27T21:52:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1751595008 bytes 2023-12-27T21:52:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1827965952 bytes 2023-12-27T21:52:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1867081728 bytes 2023-12-27T21:52:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1930483712 bytes 2023-12-27T21:52:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1952761856 bytes 2023-12-27T21:52:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 2002545664 bytes 2023-12-27T21:52:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 2049055744 bytes 2023-12-27T21:52:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 2099726336 bytes 2023-12-27T21:52:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 2135954432 bytes 2023-12-27T21:52:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 2243241984 bytes 2023-12-27T21:52:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 2290599936 bytes 2023-12-27T21:52:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 2329668608 bytes 2023-12-27T21:52:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2361190400 bytes 2023-12-27T21:52:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2378169344 bytes 2023-12-27T21:52:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2410492928 bytes 2023-12-27T21:52:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2461649920 bytes 2023-12-27T21:52:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2517940224 bytes 2023-12-27T21:52:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2538811392 bytes 2023-12-27T21:52:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2583084032 bytes 2023-12-27T21:52:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2626195456 bytes 2023-12-27T21:52:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2647424000 bytes 2023-12-27T21:52:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2686817280 bytes 2023-12-27T21:52:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2761527296 bytes 2023-12-27T21:52:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2809991168 bytes 2023-12-27T21:52:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2853556224 bytes 2023-12-27T21:52:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2882266112 bytes 2023-12-27T21:52:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2882886656 bytes 2023-12-27T21:52:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2916905984 bytes 2023-12-27T21:52:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2995828736 bytes 2023-12-27T21:52:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 3008718848 bytes 2023-12-27T21:52:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 3029724160 bytes 2023-12-27T21:52:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 3113163776 bytes 2023-12-27T21:52:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 3144546304 bytes 2023-12-27T21:52:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 3166559232 bytes 2023-12-27T21:52:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 3210833920 bytes 2023-12-27T21:52:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 3261615104 bytes 2023-12-27T21:52:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 3263301632 bytes 2023-12-27T21:52:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 3286254592 bytes 2023-12-27T21:52:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 3350155264 bytes 2023-12-27T21:52:23.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3404801024 bytes 2023-12-27T21:52:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3484657664 bytes 2023-12-27T21:52:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3520582656 bytes 2023-12-27T21:52:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3550460928 bytes 2023-12-27T21:52:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3581491200 bytes 2023-12-27T21:52:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3606568960 bytes 2023-12-27T21:52:24.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3637389312 bytes 2023-12-27T21:52:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3721261056 bytes 2023-12-27T21:52:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3823942656 bytes 2023-12-27T21:52:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3871114240 bytes 2023-12-27T21:52:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8100 IOs, 3904959488 bytes 2023-12-27T21:52:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8200 IOs, 3930072064 bytes 2023-12-27T21:52:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:52:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:52:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:52:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:52:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:52:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:52:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:52:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:52:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:52:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:52:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:52:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:52:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:52:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:52:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":17111278.93,"bytes_per_snapshot":525094912.0,"last_snapshot_bytes":513338368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703713920,"remote_snapshot_timestamp":1703713920,"replay_state":"idle"} 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.3 on smithi196 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:52:24 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:52:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:52:25 2023-12-27T21:52:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:52:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":17111278.93,"bytes_per_snapshot":525094912.0,"last_snapshot_bytes":513338368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703713920,"remote_snapshot_timestamp":1703713920,"replay_state":"idle"} 2023-12-27T21:52:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:52:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:52:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:52:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:52:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.3 on smithi196 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:52:24' 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:52:24' 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:52:24' 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:52:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:52:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:52:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:52:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:52:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:52:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:52:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:52:24' 2023-12-27T21:52:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:52:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:52:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:52:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:52:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:52:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap3 2023-12-27T21:52:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap3 2023-12-27T21:52:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap3 2023-12-27T21:52:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:52:34.922 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4599421411059149) > weight (0.25) 2023-12-27T21:52:34.923 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-27T21:52:34.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-27T21:52:34.923 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-27T21:52:34.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi196.stderr:2023-12-27T21:52:34.925+0000 7f02fd570700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 46450) UID: 1000 2023-12-27T21:52:35.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:52:43.924 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-27T21:52:43.924 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:52:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:52:43.925 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-27T21:52:43.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-27T21:52:43.925 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-27T21:52:43.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-27T21:52:43.929 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:52:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:52:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:52:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:52:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:52:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:52:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:52:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:52:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:52:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:52:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:52:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:52:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:52:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:52:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:52:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:52:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:52:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:52:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:52:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:52:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:52:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:52:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:52:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:52:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:52:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:52:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:52:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:52:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:52:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:52:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:52:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:52:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:52:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:52:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:52:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:52:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:52:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:52:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:52:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:52:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:52:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:52:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:52:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:52:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:52:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:52:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:52:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:52:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:52:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:52:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:52:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:52:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:52:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:52:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:52:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:52:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:52:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:52:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:52:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:52:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:52:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:52:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:52:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:52:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:52:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 13 2023-12-27T21:52:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:52:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:52:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:52:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:52:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:52:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=13 2023-12-27T21:52:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:52:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:52:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:52:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:52:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:52:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:52:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 13 = '' 2023-12-27T21:52:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:52:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:52:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:52:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 13 = '' 2023-12-27T21:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:52:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:52:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:52:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:52:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:52:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:52:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:53:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:53:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:53:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 13 = '' 2023-12-27T21:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:53:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:53:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:53:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:53:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:53:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=13 2023-12-27T21:53:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 13 = 13 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap3 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:53:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:53:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 6 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:53:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:52:08 2023 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:53:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap2-clean ']' 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap2-clean 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:03.130 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-27T21:53:03.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-27T21:53:03.131 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-27T21:53:03.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T21:53:03.132+0000 7f6db0e7a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 49405) UID: 1000 2023-12-27T21:53:03.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:53:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap3 2023-12-27T21:53:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap3 2023-12-27T21:53:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:53:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap3-clean 2023-12-27T21:53:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-27T21:53:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap3-clean 2023-12-27T21:53:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:53:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:53:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:53:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:53:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T21:53:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:53:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:53:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 96546816 bytes 2023-12-27T21:53:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 97161216 bytes 2023-12-27T21:53:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 164054016 bytes 2023-12-27T21:53:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 217126912 bytes 2023-12-27T21:53:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 236961792 bytes 2023-12-27T21:53:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 283218944 bytes 2023-12-27T21:53:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 362714112 bytes 2023-12-27T21:53:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 369770496 bytes 2023-12-27T21:53:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 384910336 bytes 2023-12-27T21:53:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 419298304 bytes 2023-12-27T21:53:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 462088192 bytes 2023-12-27T21:53:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 525885440 bytes 2023-12-27T21:53:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 564743168 bytes 2023-12-27T21:53:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 607755264 bytes 2023-12-27T21:53:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 654663680 bytes 2023-12-27T21:53:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 697989120 bytes 2023-12-27T21:53:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 712774656 bytes 2023-12-27T21:53:10.131 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-27T21:53:10.132 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:53:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:53:10.132 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-27T21:53:10.132 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T21:53:10.132 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T21:53:10.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T21:53:10.137 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-27T21:53:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 786207744 bytes 2023-12-27T21:53:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 844958720 bytes 2023-12-27T21:53:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 876950528 bytes 2023-12-27T21:53:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 901345280 bytes 2023-12-27T21:53:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 932128768 bytes 2023-12-27T21:53:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 979351552 bytes 2023-12-27T21:53:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1046246400 bytes 2023-12-27T21:53:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1107155968 bytes 2023-12-27T21:53:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1124549632 bytes 2023-12-27T21:53:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1137338368 bytes 2023-12-27T21:53:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1158481920 bytes 2023-12-27T21:53:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1225510912 bytes 2023-12-27T21:53:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1306221568 bytes 2023-12-27T21:53:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1350063104 bytes 2023-12-27T21:53:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1404292096 bytes 2023-12-27T21:53:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1415400448 bytes 2023-12-27T21:53:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1444210688 bytes 2023-12-27T21:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1486555136 bytes 2023-12-27T21:53:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1542791168 bytes 2023-12-27T21:53:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1558603776 bytes 2023-12-27T21:53:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1583859712 bytes 2023-12-27T21:53:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1623133184 bytes 2023-12-27T21:53:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1641889792 bytes 2023-12-27T21:53:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1665988608 bytes 2023-12-27T21:53:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1715709952 bytes 2023-12-27T21:53:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1755876352 bytes 2023-12-27T21:53:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:53:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:53:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:53:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:53:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:53:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":525094912.0,"last_snapshot_bytes":513338368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703713920,"remote_snapshot_timestamp":1703713920,"replay_state":"idle"} 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.3 on smithi196 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:52:54 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:53:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:53:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:53:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:52:55 2023-12-27T21:53:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:53:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":525094912.0,"last_snapshot_bytes":513338368,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703713920,"remote_snapshot_timestamp":1703713920,"replay_state":"idle"} 2023-12-27T21:53:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:53:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:53:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:53:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:53:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.3 on smithi196 2023-12-27T21:53:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:52:54' 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:52:54' 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:52:54' 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:53:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:53:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:53:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:53:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:53:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:53:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:53:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:52:54' 2023-12-27T21:53:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:53:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:53:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:53:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:53:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap4 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap4 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap4 2023-12-27T21:53:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:53:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:53:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:53:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:53:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:53:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:53:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:53:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:53:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:53:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:53:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:53:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:53:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:53:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:53:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:53:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:53:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:53:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:53:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:53:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:53:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:53:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:53:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:53:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:53:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:53:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:53:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:53:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:53:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:53:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:53:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:53:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:53:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:53:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:53:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:53:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:53:45.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:53:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:53:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:53:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:53:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:53:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:53:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:53:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:53:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:53:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:53:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:53:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:53:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:53:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:53:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:53:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:53:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:53:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:53:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:53:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:53:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:53:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:53:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:53:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:53:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:53:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:45.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:53:45.930 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T21:53:45.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T21:53:45.931 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-27T21:53:45.931 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T21:53:45.931 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:53:45.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T21:53:45.932+0000 7f2ac0e67700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49183) UID: 1000 2023-12-27T21:53:46.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:53:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 17 2023-12-27T21:53:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:53:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:53:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:53:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:53:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:53:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=17 2023-12-27T21:53:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:53:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:53:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:53:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:53:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:53:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:53:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:53:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:53:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:53:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:53:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:53:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 17 = '' 2023-12-27T21:53:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:53:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:53:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:53:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:53:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:53:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 17 = '' 2023-12-27T21:53:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:53:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:53:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:53:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:53:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:53:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:53:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:53:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:53:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:53:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:53:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 17 = '' 2023-12-27T21:53:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:53:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:53:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:53:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:53:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:53:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:53:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=17 2023-12-27T21:53:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 17 = 17 2023-12-27T21:53:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-27T21:53:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:53:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap4 2023-12-27T21:53:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:53:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:53:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:53:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-27T21:53:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:53:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:53:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 8 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:52:08 2023 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:53:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap3-clean ']' 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap3-clean 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:53:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:51.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:53:52.033 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T21:53:52.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T21:53:52.033 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T21:53:52.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T21:53:52.037 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-27T21:53:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap4 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap4 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:53:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:53:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:53:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:53:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap4-clean 2023-12-27T21:53:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-27T21:53:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap4-clean 2023-12-27T21:53:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:53:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:53:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:53:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:53:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:53:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:53:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:53:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=8 2023-12-27T21:53:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:53:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:53:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 62539776 bytes 2023-12-27T21:53:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 140601344 bytes 2023-12-27T21:53:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 217156608 bytes 2023-12-27T21:53:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 323598336 bytes 2023-12-27T21:53:57.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 356436992 bytes 2023-12-27T21:53:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 413416448 bytes 2023-12-27T21:53:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 451871744 bytes 2023-12-27T21:53:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 487102464 bytes 2023-12-27T21:53:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 589600768 bytes 2023-12-27T21:53:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 610057216 bytes 2023-12-27T21:53:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 632208384 bytes 2023-12-27T21:53:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 641420288 bytes 2023-12-27T21:53:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 664558592 bytes 2023-12-27T21:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 738355200 bytes 2023-12-27T21:53:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 803311616 bytes 2023-12-27T21:53:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 892048384 bytes 2023-12-27T21:53:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 912872448 bytes 2023-12-27T21:53:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 947679232 bytes 2023-12-27T21:53:59.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 988295168 bytes 2023-12-27T21:54:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1084363776 bytes 2023-12-27T21:54:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1156561920 bytes 2023-12-27T21:54:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1191444480 bytes 2023-12-27T21:54:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1213419520 bytes 2023-12-27T21:54:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1241655296 bytes 2023-12-27T21:54:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1290735616 bytes 2023-12-27T21:54:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1324381184 bytes 2023-12-27T21:54:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1357672448 bytes 2023-12-27T21:54:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1359562752 bytes 2023-12-27T21:54:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1397008384 bytes 2023-12-27T21:54:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1420233728 bytes 2023-12-27T21:54:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1428591616 bytes 2023-12-27T21:54:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1437358080 bytes 2023-12-27T21:54:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1493087232 bytes 2023-12-27T21:54:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1532640256 bytes 2023-12-27T21:54:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1576941568 bytes 2023-12-27T21:54:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1640960000 bytes 2023-12-27T21:54:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1672179712 bytes 2023-12-27T21:54:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1710519296 bytes 2023-12-27T21:54:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1797574656 bytes 2023-12-27T21:54:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1824362496 bytes 2023-12-27T21:54:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1877087232 bytes 2023-12-27T21:54:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1891832832 bytes 2023-12-27T21:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1892455424 bytes 2023-12-27T21:54:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1943078912 bytes 2023-12-27T21:54:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2054600704 bytes 2023-12-27T21:54:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2081277952 bytes 2023-12-27T21:54:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2099384320 bytes 2023-12-27T21:54:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:54:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:54:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:54:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:54:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":17597781.33,"bytes_per_snapshot":532331520.0,"last_snapshot_bytes":527933440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714026,"remote_snapshot_timestamp":1703714026,"replay_state":"idle"} 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.3 on smithi196 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:53:54 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:54:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:54:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:54:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:54:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:53:55 2023-12-27T21:54:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:54:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":17597781.33,"bytes_per_snapshot":532331520.0,"last_snapshot_bytes":527933440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714026,"remote_snapshot_timestamp":1703714026,"replay_state":"idle"} 2023-12-27T21:54:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:54:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:54:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:54:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:54:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.3 on smithi196 2023-12-27T21:54:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:53:54' 2023-12-27T21:54:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:53:54' 2023-12-27T21:54:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:54:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:53:54' 2023-12-27T21:54:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:54:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:54:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:54:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:54:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:54:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:54:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:54:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:54:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:53:54' 2023-12-27T21:54:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:54:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:54:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:54:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:54:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:54:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap5 2023-12-27T21:54:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap5 2023-12-27T21:54:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap5 2023-12-27T21:54:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:54:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:54:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:54:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:54:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:54:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:54:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:54:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:54:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:54:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:54:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:54:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:54:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:54:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:54:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:54:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:54:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:54:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:54:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:54:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:54:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:54:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:54:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:54:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:54:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:54:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:54:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:54:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:54:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:54:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:54:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:54:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:54:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:54:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:54:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:54:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:54:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:54:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:54:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:54:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:54:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:54:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:54:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:54:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:54:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:54:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:54:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:54:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:54:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:54:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:54:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:54:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:54:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:54:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:54:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:54:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:54:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:54:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:54:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:54:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:54:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:54:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:54:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:54:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:54:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:54:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:54:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:54:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:54:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:54:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:54:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:54:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:54:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:54:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:54:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:54:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:54:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:54:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:54:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:54:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:54:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 21 2023-12-27T21:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:54:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:54:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:54:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:54:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=21 2023-12-27T21:54:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:54:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:54:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:54:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:54:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:54:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:54:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:54:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:54:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:54:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:54:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:54:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 21 = '' 2023-12-27T21:54:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:54:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:54:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:54:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:54:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:54:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:54:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:54:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:54:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:54:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:54:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 21 = '' 2023-12-27T21:54:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:54:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:54:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:54:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:54:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:54:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 21 = '' 2023-12-27T21:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:54:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:54:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:54:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:54:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:54:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=21 2023-12-27T21:54:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 21 = 21 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap5 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:54:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:54:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 10 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:53:52 2023 2023-12-27T21:54:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:54:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:54:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:54:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:54:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:54:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:54:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap4-clean ']' 2023-12-27T21:54:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-27T21:54:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap4-clean 2023-12-27T21:54:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:54:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:54:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:54:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:54:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap5 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap5 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:54:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:54:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:54:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:54:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap5-clean 2023-12-27T21:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-27T21:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap5-clean 2023-12-27T21:54:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:54:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:54:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:54:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:54:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=5 2023-12-27T21:54:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:54:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:54:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 30773248 bytes 2023-12-27T21:54:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 91699200 bytes 2023-12-27T21:54:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 107446272 bytes 2023-12-27T21:54:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 190564352 bytes 2023-12-27T21:54:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 193520640 bytes 2023-12-27T21:54:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 246120448 bytes 2023-12-27T21:54:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 282137600 bytes 2023-12-27T21:54:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 339536896 bytes 2023-12-27T21:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 402686976 bytes 2023-12-27T21:54:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 425419776 bytes 2023-12-27T21:54:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 493487104 bytes 2023-12-27T21:54:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 521765888 bytes 2023-12-27T21:54:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 570686464 bytes 2023-12-27T21:54:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 612250624 bytes 2023-12-27T21:54:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 685688832 bytes 2023-12-27T21:54:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 766672896 bytes 2023-12-27T21:54:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 767283200 bytes 2023-12-27T21:54:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 836767744 bytes 2023-12-27T21:54:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 930236416 bytes 2023-12-27T21:54:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 936018944 bytes 2023-12-27T21:54:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1000180736 bytes 2023-12-27T21:54:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1043183616 bytes 2023-12-27T21:54:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1133682688 bytes 2023-12-27T21:54:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1223814144 bytes 2023-12-27T21:54:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1284408320 bytes 2023-12-27T21:54:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1294390272 bytes 2023-12-27T21:54:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1342282752 bytes 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:54:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:54:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:54:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:54:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:54:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:54:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":17597781.33,"bytes_per_snapshot":532331520.0,"last_snapshot_bytes":527933440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714026,"remote_snapshot_timestamp":1703714026,"replay_state":"idle"} 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.3 on smithi196 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:54:24 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:54:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:54:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:54:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:54:25 2023-12-27T21:54:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:54:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":17597781.33,"bytes_per_snapshot":532331520.0,"last_snapshot_bytes":527933440,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714026,"remote_snapshot_timestamp":1703714026,"replay_state":"idle"} 2023-12-27T21:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:54:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:54:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.3 on smithi196 2023-12-27T21:54:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:54:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:54:24' 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:54:24' 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:54:24' 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:54:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:54:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:54:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:54:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:54:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:54:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:54:24' 2023-12-27T21:54:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:54:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:54:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:54:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:54:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap6 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap6 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap6 2023-12-27T21:54:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:54:54.037 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T21:54:54.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T21:54:54.038 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-27T21:54:54.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T21:54:54.039+0000 7f47e4aeb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 51446) UID: 1000 2023-12-27T21:54:54.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:55:00.039 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T21:55:00.039 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:55:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:55:00.039 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T21:55:00.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T21:55:00.040 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T21:55:00.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T21:55:00.043 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-12-27T21:55:02.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.49862997385298535) > weight (0.25) 2023-12-27T21:55:02.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.36737765821056223) > weight (0.25) 2023-12-27T21:55:02.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5503324523814463) > weight (0.25) 2023-12-27T21:55:02.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7159749147487415) > weight (0.25) 2023-12-27T21:55:02.139 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-27T21:55:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:55:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:55:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:55:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:55:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:55:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:55:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:55:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:55:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:55:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:55:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:55:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:55:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:55:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:55:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:55:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:55:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:55:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:55:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:55:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:55:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:55:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:55:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:55:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:55:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:55:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:55:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:55:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:55:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:55:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:55:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:55:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:55:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:55:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:55:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:55:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:55:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:55:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:55:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:55:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:55:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:55:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:55:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:55:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:55:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:55:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:55:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:55:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:55:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:55:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:55:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:55:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:55:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:55:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:55:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:55:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:55:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:55:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:55:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:55:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:55:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:55:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:55:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:55:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:55:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 25 2023-12-27T21:55:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:55:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:55:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:55:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:55:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:55:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:55:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=25 2023-12-27T21:55:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:55:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:55:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:55:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:55:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:55:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:55:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:55:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:55:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:55:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:55:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:55:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 25 = '' 2023-12-27T21:55:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:55:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:55:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:55:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:55:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:55:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:55:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:55:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:55:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:55:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:55:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 25 = '' 2023-12-27T21:55:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:55:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:55:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:55:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:55:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:55:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:55:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:55:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:55:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:55:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 25 = '' 2023-12-27T21:55:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:55:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:55:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:55:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:55:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:55:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:55:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=25 2023-12-27T21:55:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 25 = 25 2023-12-27T21:55:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap6 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:55:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:55:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-27T21:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:55:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 12 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:53:52 2023 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:55:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap5-clean ']' 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap5-clean 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:55:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:55:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:55:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap6 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap6 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:55:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:55:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:55:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:55:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap6-clean 2023-12-27T21:55:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-27T21:55:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:55:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap6-clean 2023-12-27T21:55:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:55:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:55:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:55:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T21:55:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:55:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:55:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 38097920 bytes 2023-12-27T21:55:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 86626304 bytes 2023-12-27T21:55:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 125167616 bytes 2023-12-27T21:55:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 201179136 bytes 2023-12-27T21:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 221522944 bytes 2023-12-27T21:55:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 222147584 bytes 2023-12-27T21:55:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 245344256 bytes 2023-12-27T21:55:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 247507968 bytes 2023-12-27T21:55:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 276916224 bytes 2023-12-27T21:55:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 315826176 bytes 2023-12-27T21:55:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 346469376 bytes 2023-12-27T21:55:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 405482496 bytes 2023-12-27T21:55:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 433532928 bytes 2023-12-27T21:55:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 460520448 bytes 2023-12-27T21:55:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 515677184 bytes 2023-12-27T21:55:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 542247936 bytes 2023-12-27T21:55:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 600672256 bytes 2023-12-27T21:55:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 695209984 bytes 2023-12-27T21:55:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 732731392 bytes 2023-12-27T21:55:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 760991744 bytes 2023-12-27T21:55:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 813980672 bytes 2023-12-27T21:55:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 823706624 bytes 2023-12-27T21:55:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 833455104 bytes 2023-12-27T21:55:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 882860032 bytes 2023-12-27T21:55:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 979442688 bytes 2023-12-27T21:55:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 988298240 bytes 2023-12-27T21:55:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1033065472 bytes 2023-12-27T21:55:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1069954048 bytes 2023-12-27T21:55:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1070560256 bytes 2023-12-27T21:55:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1122693120 bytes 2023-12-27T21:55:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1188169728 bytes 2023-12-27T21:55:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1226507264 bytes 2023-12-27T21:55:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1238268928 bytes 2023-12-27T21:55:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1240554496 bytes 2023-12-27T21:55:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1285020672 bytes 2023-12-27T21:55:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1323272192 bytes 2023-12-27T21:55:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1323895808 bytes 2023-12-27T21:55:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1338614784 bytes 2023-12-27T21:55:41.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1372621824 bytes 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:55:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:55:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:55:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:55:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:55:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530997248.0,"last_snapshot_bytes":534061056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714077,"remote_snapshot_timestamp":1703714077,"replay_state":"idle"} 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.3 on smithi196 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:55:24 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:55:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:55:25 2023-12-27T21:55:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:55:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530997248.0,"last_snapshot_bytes":534061056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714077,"remote_snapshot_timestamp":1703714077,"replay_state":"idle"} 2023-12-27T21:55:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:55:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:55:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:55:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:55:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.3 on smithi196 2023-12-27T21:55:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:55:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:55:24' 2023-12-27T21:55:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:55:24' 2023-12-27T21:55:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:55:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:55:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:55:24' 2023-12-27T21:55:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:55:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:55:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:55:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:55:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:55:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:55:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:55:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:55:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:55:24' 2023-12-27T21:55:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:55:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:55:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:55:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:55:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:55:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:55:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap7 2023-12-27T21:55:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap7 2023-12-27T21:55:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:55:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:55:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:55:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap7 2023-12-27T21:55:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:56:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:56:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:56:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:56:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:56:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:56:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:56:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:56:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:56:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:56:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:56:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:56:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:56:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:56:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:56:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:56:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:56:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:56:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:56:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:56:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:56:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:56:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:56:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:56:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:56:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:56:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:56:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:56:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:56:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:56:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:56:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:56:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:56:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:56:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:56:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:56:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:56:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:56:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:56:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:56:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:56:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:56:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:56:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:56:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:56:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:56:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:56:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:56:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:56:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:56:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:56:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:56:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:56:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:56:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:56:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:56:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:56:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:56:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T21:56:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T21:56:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T21:56:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:56:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:56:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T21:56:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-27T21:56:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:56:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:56:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:56:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:56:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:56:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:56:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:56:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:56:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:56:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:56:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:56:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:56:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:56:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:56:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 29 2023-12-27T21:56:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:56:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:56:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:56:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:56:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:56:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:56:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=29 2023-12-27T21:56:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:56:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:56:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:56:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:56:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:56:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:56:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 29 = '' 2023-12-27T21:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:56:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:56:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:56:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:56:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:56:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:56:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:56:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:56:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:56:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 29 = '' 2023-12-27T21:56:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:56:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:56:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:56:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:56:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:56:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:56:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:56:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:56:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:56:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:56:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 29 = '' 2023-12-27T21:56:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:56:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:56:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:56:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:56:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:56:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:56:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=29 2023-12-27T21:56:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 29 = 29 2023-12-27T21:56:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-27T21:56:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap7 2023-12-27T21:56:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:56:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:56:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:56:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-27T21:56:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 14 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:55:31 2023 2023-12-27T21:56:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:56:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:56:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:56:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:56:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:56:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:56:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap6-clean ']' 2023-12-27T21:56:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-27T21:56:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap6-clean 2023-12-27T21:56:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:56:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:56:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:56:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:56:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:21.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.38196517477274705) > weight (0.25) 2023-12-27T21:56:21.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5996678515587097) > weight (0.25) 2023-12-27T21:56:21.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9696717129618379) > weight (0.25) 2023-12-27T21:56:21.044 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-27T21:56:21.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-27T21:56:21.045 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-27T21:56:21.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi196.stderr:2023-12-27T21:56:21.046+0000 7f36a9f9f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 46485) UID: 1000 2023-12-27T21:56:21.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:56:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:56:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap7 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap7 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:56:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:56:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:56:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap7-clean 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap7-clean 2023-12-27T21:56:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:56:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:56:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:56:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:56:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:56:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T21:56:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:56:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:56:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 52350976 bytes 2023-12-27T21:56:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 108651520 bytes 2023-12-27T21:56:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 144915456 bytes 2023-12-27T21:56:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 177385472 bytes 2023-12-27T21:56:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 221444096 bytes 2023-12-27T21:56:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 277911552 bytes 2023-12-27T21:56:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 332076032 bytes 2023-12-27T21:56:26.045 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-27T21:56:26.045 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:56:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:56:26.046 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-27T21:56:26.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-27T21:56:26.046 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-27T21:56:26.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-27T21:56:26.049 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-27T21:56:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 390128640 bytes 2023-12-27T21:56:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 392255488 bytes 2023-12-27T21:56:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 506320896 bytes 2023-12-27T21:56:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 584411136 bytes 2023-12-27T21:56:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 610289664 bytes 2023-12-27T21:56:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 663582720 bytes 2023-12-27T21:56:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 705902592 bytes 2023-12-27T21:56:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 756517888 bytes 2023-12-27T21:56:27.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 835126272 bytes 2023-12-27T21:56:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 850163712 bytes 2023-12-27T21:56:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 919182336 bytes 2023-12-27T21:56:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 978387968 bytes 2023-12-27T21:56:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1052089344 bytes 2023-12-27T21:56:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1060981760 bytes 2023-12-27T21:56:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1101622272 bytes 2023-12-27T21:56:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1137756160 bytes 2023-12-27T21:56:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1202212864 bytes 2023-12-27T21:56:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1268857856 bytes 2023-12-27T21:56:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1315654656 bytes 2023-12-27T21:56:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1368453120 bytes 2023-12-27T21:56:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1438825472 bytes 2023-12-27T21:56:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1448519680 bytes 2023-12-27T21:56:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1470937088 bytes 2023-12-27T21:56:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1521302528 bytes 2023-12-27T21:56:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1563998208 bytes 2023-12-27T21:56:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1595815936 bytes 2023-12-27T21:56:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1605604352 bytes 2023-12-27T21:56:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1608915968 bytes 2023-12-27T21:56:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1681462272 bytes 2023-12-27T21:56:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1744894976 bytes 2023-12-27T21:56:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1778650112 bytes 2023-12-27T21:56:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1813313536 bytes 2023-12-27T21:56:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1831747584 bytes 2023-12-27T21:56:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1851800576 bytes 2023-12-27T21:56:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1881415680 bytes 2023-12-27T21:56:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1954571264 bytes 2023-12-27T21:56:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1994766336 bytes 2023-12-27T21:56:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2065860608 bytes 2023-12-27T21:56:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2119394304 bytes 2023-12-27T21:56:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2137050112 bytes 2023-12-27T21:56:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2200424448 bytes 2023-12-27T21:56:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2219765760 bytes 2023-12-27T21:56:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2295907328 bytes 2023-12-27T21:56:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2326406144 bytes 2023-12-27T21:56:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2351428608 bytes 2023-12-27T21:56:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2400598016 bytes 2023-12-27T21:56:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2411104256 bytes 2023-12-27T21:56:34.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2428673024 bytes 2023-12-27T21:56:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2481624064 bytes 2023-12-27T21:56:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2544462848 bytes 2023-12-27T21:56:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2564197376 bytes 2023-12-27T21:56:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2604288000 bytes 2023-12-27T21:56:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2735745024 bytes 2023-12-27T21:56:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2763091968 bytes 2023-12-27T21:56:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2833299456 bytes 2023-12-27T21:56:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2883658752 bytes 2023-12-27T21:56:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2884282368 bytes 2023-12-27T21:56:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2884911104 bytes 2023-12-27T21:56:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2952958976 bytes 2023-12-27T21:56:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2990615552 bytes 2023-12-27T21:56:36.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 3010586624 bytes 2023-12-27T21:56:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 3076890624 bytes 2023-12-27T21:56:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 3118527488 bytes 2023-12-27T21:56:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3127038976 bytes 2023-12-27T21:56:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3150665728 bytes 2023-12-27T21:56:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3181239296 bytes 2023-12-27T21:56:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3194193920 bytes 2023-12-27T21:56:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3242454016 bytes 2023-12-27T21:56:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3297410048 bytes 2023-12-27T21:56:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3366093824 bytes 2023-12-27T21:56:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3484876800 bytes 2023-12-27T21:56:38.140 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7822086450807705) > weight (0.25) 2023-12-27T21:56:38.140 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-27T21:56:38.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-27T21:56:38.140 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-27T21:56:38.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T21:56:38.141+0000 7f8efbe6c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 46526) UID: 1000 2023-12-27T21:56:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3521838080 bytes 2023-12-27T21:56:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3549473792 bytes 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:56:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:56:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:56:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:56:38.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:56:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:56:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:56:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:56:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1703714174,"remote_snapshot_timestamp":1703714174,"replay_state":"idle"} 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.2 on smithi196 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:56:25 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: down+stopped 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:56:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:56:25 2023-12-27T21:56:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:56:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1703714174,"remote_snapshot_timestamp":1703714174,"replay_state":"idle"} 2023-12-27T21:56:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:56:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:56:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:56:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:56:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.2 on smithi196 2023-12-27T21:56:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:56:25' 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:56:25' 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:56:25' 2023-12-27T21:56:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:56:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:56:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:56:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:56:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:56:25' 2023-12-27T21:56:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:56:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:56:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:56:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:56:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap8 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap8 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap8 2023-12-27T21:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:56:43.142 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-27T21:56:43.142 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:56:43.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:56:43.142 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-27T21:56:43.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-27T21:56:43.142 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-27T21:56:43.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-27T21:56:43.145 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:57:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:57:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:57:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:57:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:57:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:57:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:57:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:57:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:57:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:57:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:57:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:57:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:57:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:57:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:57:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:57:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:57:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:57:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:57:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:57:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:57:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:57:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:57:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:57:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:57:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:57:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:57:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:57:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:57:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:57:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:57:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:57:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:57:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:57:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:57:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:57:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:57:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:57:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:57:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:57:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:57:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:57:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:57:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:57:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:57:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:57:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:57:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:57:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:57:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:57:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:57:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:57:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:57:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:57:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:57:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:57:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:57:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:57:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:57:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:57:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:57:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:57:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:57:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:57:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:57:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:57:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:57:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:57:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:57:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:57:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:57:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:57:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:57:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:57:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:57:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:57:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:57:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:57:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:57:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 33 2023-12-27T21:57:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:57:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:57:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:57:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:57:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:57:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:57:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=33 2023-12-27T21:57:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:57:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:57:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:57:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:57:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:57:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:57:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:57:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:57:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:57:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:57:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:57:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 33 = '' 2023-12-27T21:57:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:57:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:57:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:57:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:57:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:57:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:57:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:57:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:57:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:57:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 33 = '' 2023-12-27T21:57:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:57:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:57:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:57:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:57:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:57:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:57:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 33 = '' 2023-12-27T21:57:11.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:57:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:57:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:57:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:57:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:57:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:57:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=33 2023-12-27T21:57:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 33 = 33 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap8 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:57:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:57:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-27T21:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 16 2023-12-27T21:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:57:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:55:31 2023 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:57:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap7-clean ']' 2023-12-27T21:57:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-27T21:57:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap7-clean 2023-12-27T21:57:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:57:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:57:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:57:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:57:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:57:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap8 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap8 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:57:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:57:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:57:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:57:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap8-clean 2023-12-27T21:57:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-27T21:57:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:57:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap8-clean 2023-12-27T21:57:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:57:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:57:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:57:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:57:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:57:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T21:57:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:57:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:57:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 17523712 bytes 2023-12-27T21:57:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 54484992 bytes 2023-12-27T21:57:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 74438656 bytes 2023-12-27T21:57:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 94777344 bytes 2023-12-27T21:57:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 148508672 bytes 2023-12-27T21:57:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 161455104 bytes 2023-12-27T21:57:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 241344512 bytes 2023-12-27T21:57:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 308129792 bytes 2023-12-27T21:57:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 336011264 bytes 2023-12-27T21:57:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 372005888 bytes 2023-12-27T21:57:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 387539968 bytes 2023-12-27T21:57:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 444478464 bytes 2023-12-27T21:57:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 473382912 bytes 2023-12-27T21:57:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 518629376 bytes 2023-12-27T21:57:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 553957376 bytes 2023-12-27T21:57:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 626391040 bytes 2023-12-27T21:57:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 685253632 bytes 2023-12-27T21:57:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 710683648 bytes 2023-12-27T21:57:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 736693248 bytes 2023-12-27T21:57:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 793689088 bytes 2023-12-27T21:57:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 822306816 bytes 2023-12-27T21:57:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 841665536 bytes 2023-12-27T21:57:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 861030400 bytes 2023-12-27T21:57:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 950667264 bytes 2023-12-27T21:57:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 998903808 bytes 2023-12-27T21:57:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1078802432 bytes 2023-12-27T21:57:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1155981312 bytes 2023-12-27T21:57:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1169570816 bytes 2023-12-27T21:57:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1203248128 bytes 2023-12-27T21:57:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1224128512 bytes 2023-12-27T21:57:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1265196032 bytes 2023-12-27T21:57:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1289345024 bytes 2023-12-27T21:57:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1305893888 bytes 2023-12-27T21:57:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1355419648 bytes 2023-12-27T21:57:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1362444288 bytes 2023-12-27T21:57:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1410380800 bytes 2023-12-27T21:57:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1469661184 bytes 2023-12-27T21:57:24.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1493705728 bytes 2023-12-27T21:57:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1528733696 bytes 2023-12-27T21:57:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1583730688 bytes 2023-12-27T21:57:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1626987520 bytes 2023-12-27T21:57:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1636893696 bytes 2023-12-27T21:57:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1643487232 bytes 2023-12-27T21:57:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1694782464 bytes 2023-12-27T21:57:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1727321088 bytes 2023-12-27T21:57:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1776957440 bytes 2023-12-27T21:57:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1796540416 bytes 2023-12-27T21:57:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1832013824 bytes 2023-12-27T21:57:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1907997696 bytes 2023-12-27T21:57:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1915809792 bytes 2023-12-27T21:57:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 1988953088 bytes 2023-12-27T21:57:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 1993280512 bytes 2023-12-27T21:57:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2022813696 bytes 2023-12-27T21:57:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2083634176 bytes 2023-12-27T21:57:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2084240384 bytes 2023-12-27T21:57:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2116116480 bytes 2023-12-27T21:57:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2171413504 bytes 2023-12-27T21:57:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2198432768 bytes 2023-12-27T21:57:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2273345536 bytes 2023-12-27T21:57:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2297529344 bytes 2023-12-27T21:57:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2330907648 bytes 2023-12-27T21:57:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2358619136 bytes 2023-12-27T21:57:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2397907968 bytes 2023-12-27T21:57:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2413248512 bytes 2023-12-27T21:57:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2446929920 bytes 2023-12-27T21:57:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2507194368 bytes 2023-12-27T21:57:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2574759936 bytes 2023-12-27T21:57:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2659445760 bytes 2023-12-27T21:57:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2702267392 bytes 2023-12-27T21:57:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 2750074880 bytes 2023-12-27T21:57:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 2766768128 bytes 2023-12-27T21:57:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 2838061056 bytes 2023-12-27T21:57:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 2880219136 bytes 2023-12-27T21:57:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 2917263360 bytes 2023-12-27T21:57:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 2952949760 bytes 2023-12-27T21:57:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3001355264 bytes 2023-12-27T21:57:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3002006528 bytes 2023-12-27T21:57:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3022489600 bytes 2023-12-27T21:57:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3136212992 bytes 2023-12-27T21:57:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3136825344 bytes 2023-12-27T21:57:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8100 IOs, 3141091328 bytes 2023-12-27T21:57:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8200 IOs, 3194807296 bytes 2023-12-27T21:57:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8300 IOs, 3255123968 bytes 2023-12-27T21:57:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8400 IOs, 3301912576 bytes 2023-12-27T21:57:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:57:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:57:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:57:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:57:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:57:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:57:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:57:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:57:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:57:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:57:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:57:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:57:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:57:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:57:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:57:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:57:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:57:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:57:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:57:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":17856136.53,"bytes_per_snapshot":535684096.0,"last_snapshot_bytes":535684096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714228,"remote_snapshot_timestamp":1703714228,"replay_state":"idle"} 2023-12-27T21:57:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.2 on smithi196 2023-12-27T21:57:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:57:24 2023-12-27T21:57:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:57:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:57:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:57:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:57:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:57:10 2023-12-27T21:57:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:57:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":17856136.53,"bytes_per_snapshot":535684096.0,"last_snapshot_bytes":535684096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714228,"remote_snapshot_timestamp":1703714228,"replay_state":"idle"} 2023-12-27T21:57:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:57:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:57:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:57:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:57:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.2 on smithi196 2023-12-27T21:57:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:57:24' 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:57:24' 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:57:24' 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:57:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:57:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:57:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:57:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:57:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:57:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:57:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:57:24' 2023-12-27T21:57:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:57:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:57:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:57:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:57:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:57:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:57:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap9 2023-12-27T21:57:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap9 2023-12-27T21:57:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:57:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:57:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:57:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap9 2023-12-27T21:57:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:57:56.050 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T21:57:56.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T21:57:56.050 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T21:57:56.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T21:57:56.052+0000 7f11bdcd5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 52198) UID: 1000 2023-12-27T21:57:56.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:58:00.051 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T21:58:00.052 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:58:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:58:00.052 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T21:58:00.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T21:58:00.052 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T21:58:00.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T21:58:00.056 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-27T21:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:58:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:58:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:58:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:58:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:58:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:58:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:58:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:58:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:58:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:58:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:58:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:58:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:58:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:58:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:58:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:58:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:58:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:58:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:58:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:58:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:58:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:58:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:58:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:58:04.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:58:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:58:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:58:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:58:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:58:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:58:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:58:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:58:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:58:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:58:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:58:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:58:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:58:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:58:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:58:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:58:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:58:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:58:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:58:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:58:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:58:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:58:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:58:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:58:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:58:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:58:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:58:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:58:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 37 2023-12-27T21:58:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:58:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:58:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:58:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:58:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=37 2023-12-27T21:58:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:58:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:58:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:58:06.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 37 = '' 2023-12-27T21:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:58:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:58:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:58:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:58:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:58:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 37 = '' 2023-12-27T21:58:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:58:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:58:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:58:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:58:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:58:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 37 = '' 2023-12-27T21:58:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:58:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=37 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 37 = 37 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap9 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:58:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:58:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-27T21:58:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:58:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:58:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 18 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:57:14 2023 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:58:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap8-clean ']' 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap8-clean 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:58:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:58:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:58:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:58:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap9 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap9 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:58:13.146 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8895145960361763) > weight (0.25) 2023-12-27T21:58:13.146 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-27T21:58:13.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-27T21:58:13.147 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T21:58:13.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T21:58:13.147+0000 7fc227e8b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 53262) UID: 1000 2023-12-27T21:58:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:58:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:13.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T21:58:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:58:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:58:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap9-clean 2023-12-27T21:58:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-27T21:58:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap9-clean 2023-12-27T21:58:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:58:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:58:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror test 2023-12-27T21:58:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T21:58:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T21:58:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T21:58:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-27T21:58:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 27290624 bytes 2023-12-27T21:58:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 72863744 bytes 2023-12-27T21:58:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 100356096 bytes 2023-12-27T21:58:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 182173696 bytes 2023-12-27T21:58:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 247434240 bytes 2023-12-27T21:58:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 327721984 bytes 2023-12-27T21:58:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 346903552 bytes 2023-12-27T21:58:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 357183488 bytes 2023-12-27T21:58:17.147 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-27T21:58:17.147 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T21:58:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T21:58:17.148 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-27T21:58:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-27T21:58:17.148 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-27T21:58:17.151 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-27T21:58:17.151 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-27T21:58:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 401116160 bytes 2023-12-27T21:58:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 498293760 bytes 2023-12-27T21:58:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 539710464 bytes 2023-12-27T21:58:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 599221248 bytes 2023-12-27T21:58:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 629016576 bytes 2023-12-27T21:58:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 659482624 bytes 2023-12-27T21:58:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 747953152 bytes 2023-12-27T21:58:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 757485568 bytes 2023-12-27T21:58:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 771095552 bytes 2023-12-27T21:58:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 786284544 bytes 2023-12-27T21:58:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 823861248 bytes 2023-12-27T21:58:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 871472128 bytes 2023-12-27T21:58:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 896360448 bytes 2023-12-27T21:58:19.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 949212160 bytes 2023-12-27T21:58:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1021831168 bytes 2023-12-27T21:58:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1029916672 bytes 2023-12-27T21:58:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1085001728 bytes 2023-12-27T21:58:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1105351680 bytes 2023-12-27T21:58:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1147875328 bytes 2023-12-27T21:58:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1186305024 bytes 2023-12-27T21:58:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1205199872 bytes 2023-12-27T21:58:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1205799936 bytes 2023-12-27T21:58:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1260478464 bytes 2023-12-27T21:58:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1300159488 bytes 2023-12-27T21:58:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1345334272 bytes 2023-12-27T21:58:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1373019136 bytes 2023-12-27T21:58:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1400878080 bytes 2023-12-27T21:58:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1443535872 bytes 2023-12-27T21:58:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1484427264 bytes 2023-12-27T21:58:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1506656256 bytes 2023-12-27T21:58:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1508415488 bytes 2023-12-27T21:58:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1572023296 bytes 2023-12-27T21:58:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1589101568 bytes 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:58:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:58:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-27T21:58:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local description_pattern= 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local service_pattern= 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:58:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ CEPH_ARGS= 2023-12-27T21:58:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-27T21:58:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:test: 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: global_id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+replaying 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535684096.0,"last_snapshot_bytes":535684096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714228,"remote_snapshot_timestamp":1703714228,"replay_state":"idle"} 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: service: mirror.2 on smithi196 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:57:54 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: peer_sites: 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: name: cluster2 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: state: up+stopped 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: description: local image is primary 2023-12-27T21:58:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: last_update: 2023-12-27 21:58:10 2023-12-27T21:58:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: state: up+replaying 2023-12-27T21:58:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^ description: .*' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535684096.0,"last_snapshot_bytes":535684096,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1703714228,"remote_snapshot_timestamp":1703714228,"replay_state":"idle"} 2023-12-27T21:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T21:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep '^up+' 2023-12-27T21:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: service: mirror.2 on smithi196 2023-12-27T21:58:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.mirror_status 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'last_update=2023-12-27 21:57:54' 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-27 21:57:54' 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state_pattern=up+replaying 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local 'prev_last_update=2023-12-27 21:57:54' 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror.mirror_status 2023-12-27T21:58:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:58:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror.mirror_status 2023-12-27T21:58:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-27T21:58:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T21:58:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_update state 2023-12-27T21:58:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-27T21:58:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ last_update='2023-12-27 21:57:54' 2023-12-27T21:58:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-27T21:58:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=up+replaying 2023-12-27T21:58:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo up+replaying 2023-12-27T21:58:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep up+replaying 2023-12-27T21:58:23.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:up+replaying 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap10 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap10 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap10 2023-12-27T21:58:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-27T21:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:58:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:58:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:58:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:58:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:58:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:58:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:58:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:58:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:58:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:58:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:58:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:58:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:58:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:58:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:58:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:58:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:58:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:58:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:58:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:58:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:58:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:58:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:58:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:58:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:58:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:58:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:58:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:58:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:58:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:58:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:58:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:58:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:58:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:58:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:58:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:58:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:58:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:58:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:58:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:58:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:58:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:58:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:58:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:58:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:58:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:58:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:58:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:58:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:58:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:58:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:58:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:58:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:58:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:58:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:58:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:58:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 41 2023-12-27T21:58:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:58:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:58:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:58:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:58:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:58:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=41 2023-12-27T21:58:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:58:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:58:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:58:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:58:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 41 = '' 2023-12-27T21:58:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:58:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:58:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:58:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:58:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:58:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 41 = '' 2023-12-27T21:58:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T21:58:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:58:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:58:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:58:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:58:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:58:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 41 = '' 2023-12-27T21:58:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:58:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T21:58:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:58:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:58:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:58:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:58:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=41 2023-12-27T21:58:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 41 = 41 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap10 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:58:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:59:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-27T21:59:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:59:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 20 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:59:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:59:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:59:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:57:14 2023 2023-12-27T21:59:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:59:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:59:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:59:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:59:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:59:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snap9-clean ']' 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap9-clean 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap10 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap10 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:03.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ clean_snap_name=snap10-clean 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap10-clean 2023-12-27T21:59:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-27T21:59:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-27T21:59:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:59:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-27T21:59:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:59:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T21:59:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:59:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:59:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:59:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-27T21:59:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:59:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T21:59:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T21:59:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T21:59:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:59:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T21:59:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:59:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:59:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:59:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T21:59:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T21:59:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:59:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:59:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T21:59:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T21:59:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T21:59:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:59:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:59:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T21:59:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-27T21:59:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:59:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:59:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:59:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:59:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:59:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:59:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T21:59:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T21:59:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T21:59:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:59:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:59:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T21:59:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-27T21:59:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-27T21:59:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T21:59:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T21:59:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T21:59:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T21:59:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T21:59:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T21:59:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T21:59:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T21:59:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T21:59:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:59:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:59:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T21:59:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-27T21:59:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T21:59:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T21:59:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T21:59:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T21:59:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-test.status 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-test.status 2023-12-27T21:59:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:59:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:59:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-test.status 2023-12-27T21:59:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-test.status 2023-12-27T21:59:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T21:59:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:59:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-27T21:59:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-27T21:59:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 44 2023-12-27T21:59:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:59:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.status 2023-12-27T21:59:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-27T21:59:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T21:59:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=44 2023-12-27T21:59:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:59:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:59:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T21:59:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:59:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:59:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:59:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:59:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:59:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T21:59:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:59:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T21:59:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T21:59:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T21:59:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T21:59:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 44 = '' 2023-12-27T21:59:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T21:59:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T21:59:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:59:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:59:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.status 2023-12-27T21:59:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-27T21:59:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T21:59:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=44 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 44 = 44 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap10-clean 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T21:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T21:59:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'test': 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 21 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 10956c066f81 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.10956c066f81 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T21:59:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 21:59:00 2023 2023-12-27T21:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 21:50:27 2023 2023-12-27T21:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T21:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T21:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T21:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d75ae850-0356-482d-b2a1-cae31e42567f 2023-12-27T21:59:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T21:59:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 10 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap1 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap1 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap1 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap1-clean 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap1-clean 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap2 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap2 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap2 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap2-clean 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap2-clean 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap3 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap3 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap3 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap3-clean 2023-12-27T21:59:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap3-clean 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap4 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap4 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap4 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap4-clean 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap4-clean 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap5 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap5 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap5 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap5-clean 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap5-clean 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap6 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap6 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap6 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap6-clean 2023-12-27T21:59:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-27T21:59:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap6-clean 2023-12-27T21:59:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap7 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap7 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap7 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap7-clean 2023-12-27T21:59:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap7-clean 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap8 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap8 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap8 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap8-clean 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap8-clean 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap9 2023-12-27T21:59:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap9 2023-12-27T21:59:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap9 2023-12-27T21:59:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap9-clean 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap9-clean 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:42.057 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6516680220021069) > weight (0.25) 2023-12-27T21:59:42.057 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6914158981980151) > weight (0.25) 2023-12-27T21:59:42.057 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3100111339265582) > weight (0.25) 2023-12-27T21:59:42.057 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8840255807430316) > weight (0.25) 2023-12-27T21:59:42.057 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-27T21:59:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap10 2023-12-27T21:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap10 2023-12-27T21:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap10 2023-12-27T21:59:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap10-clean 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap10-clean 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T21:59:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export 2023-12-27T21:59:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-27T21:59:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-test.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-test.export 2023-12-27T21:59:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T21:59:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 10 2023-12-27T21:59:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap1 2023-12-27T21:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-27T21:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap1 2023-12-27T21:59:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-27T21:59:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap1-clean 2023-12-27T21:59:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-27T21:59:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap1-clean 2023-12-27T21:59:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-27T21:59:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap2 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap2 2023-12-27T21:59:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-27T21:59:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap2-clean 2023-12-27T21:59:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-27T21:59:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap2-clean 2023-12-27T21:59:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-27T21:59:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap3 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap3 2023-12-27T21:59:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-27T21:59:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap3-clean 2023-12-27T21:59:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-27T21:59:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap3-clean 2023-12-27T21:59:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-27T21:59:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap4 2023-12-27T21:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-27T21:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap4 2023-12-27T21:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-27T21:59:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap4-clean 2023-12-27T21:59:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-27T21:59:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap4-clean 2023-12-27T21:59:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-27T21:59:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap5 2023-12-27T21:59:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-27T21:59:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap5 2023-12-27T21:59:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-27T21:59:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap5-clean 2023-12-27T21:59:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-27T21:59:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap5-clean 2023-12-27T21:59:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-27T21:59:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap6 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap6 2023-12-27T21:59:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-27T21:59:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap6-clean 2023-12-27T21:59:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-27T21:59:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap6-clean 2023-12-27T21:59:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-27T21:59:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T21:59:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap7 2023-12-27T21:59:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-27T21:59:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap7 2023-12-27T21:59:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-27T21:59:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T21:59:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap7-clean 2023-12-27T21:59:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-27T21:59:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T21:59:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T21:59:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T21:59:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap7-clean 2023-12-27T21:59:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-27T22:00:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap8 2023-12-27T22:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-27T22:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap8 2023-12-27T22:00:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-27T22:00:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:00:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap8-clean 2023-12-27T22:00:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-27T22:00:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap8-clean 2023-12-27T22:00:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-27T22:00:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:00:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:00:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap9 2023-12-27T22:00:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-27T22:00:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap9 2023-12-27T22:00:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-27T22:00:03.151 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7728014514433571) > weight (0.25) 2023-12-27T22:00:03.151 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.720439979043696) > weight (0.25) 2023-12-27T22:00:03.151 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.35679227105181) > weight (0.25) 2023-12-27T22:00:03.151 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7592185220160866) > weight (0.25) 2023-12-27T22:00:03.151 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-27T22:00:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:00:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap9-clean 2023-12-27T22:00:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-27T22:00:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap9-clean 2023-12-27T22:00:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-27T22:00:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:00:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:00:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap10 2023-12-27T22:00:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-27T22:00:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap10 2023-12-27T22:00:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-27T22:00:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:00:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap10-clean 2023-12-27T22:00:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-27T22:00:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap10-clean 2023-12-27T22:00:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-27T22:00:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:00:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror test 2023-12-27T22:00:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:00:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:00:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror test 2023-12-27T22:00:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-27T22:00:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-27T22:00:06.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T22:00:06.518+0000 7f6920b8c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55add3d80340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:00:06.519 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T22:00:06.518+0000 7f6927399700 -1 rbd::mirror::ImageReplayer: 0x55adcfcf8280 [3/d75ae850-0356-482d-b2a1-cae31e42567f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:00:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:00:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:00:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-27T22:00:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:00:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror test 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=test 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-27T22:00:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:00:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-27T22:00:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:00:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:00:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=test 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror test 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:00:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:00:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=test 2023-12-27T22:00:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-27T22:00:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:00:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-27T22:00:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:00:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:00:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:00:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:00:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ testlog 'TEST: create many images' 2023-12-27T22:00:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee -a /tmp/tmp.yuhsqMaECB/rbd-mirror.test.log 2023-12-27T22:00:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ date '+%F %T' 2023-12-27T22:00:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo 2023-12-27 22:00:08 TEST: create many images 2023-12-27T22:00:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-27 22:00:08 TEST: create many images 2023-12-27T22:00:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ snap_name=snap 2023-12-27T22:00:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 50 2023-12-27T22:00:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:00:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_1 2023-12-27T22:00:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-27T22:00:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:00:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:00:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:00:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-27T22:00:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:00:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-27T22:00:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:00:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-27T22:00:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:00:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_1': 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 125cfcca2554 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.125cfcca2554 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:00:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:08 2023 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:08 2023 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:08 2023 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 42426da1-e209-4023-8d5f-43ff971b1cd6 2023-12-27T22:00:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:00:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:00:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-27T22:00:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:00:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:00:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T22:00:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:00:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:00:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 10800128 bytes 2023-12-27T22:00:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 96221184 bytes 2023-12-27T22:00:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 160962560 bytes 2023-12-27T22:00:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 180039680 bytes 2023-12-27T22:00:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 213259264 bytes 2023-12-27T22:00:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 214093824 bytes 2023-12-27T22:00:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 325099520 bytes 2023-12-27T22:00:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 391271424 bytes 2023-12-27T22:00:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 436208640 bytes 2023-12-27T22:00:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 511324160 bytes 2023-12-27T22:00:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 586691584 bytes 2023-12-27T22:00:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 598650880 bytes 2023-12-27T22:00:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 616147968 bytes 2023-12-27T22:00:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 644700160 bytes 2023-12-27T22:00:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 717822976 bytes 2023-12-27T22:00:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 798318592 bytes 2023-12-27T22:00:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 838784000 bytes 2023-12-27T22:00:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 875554816 bytes 2023-12-27T22:00:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 918550528 bytes 2023-12-27T22:00:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 938896384 bytes 2023-12-27T22:00:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1002988544 bytes 2023-12-27T22:00:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1034317824 bytes 2023-12-27T22:00:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1069491200 bytes 2023-12-27T22:00:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1118411776 bytes 2023-12-27T22:00:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1135984640 bytes 2023-12-27T22:00:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1168121856 bytes 2023-12-27T22:00:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1197877248 bytes 2023-12-27T22:00:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1250832384 bytes 2023-12-27T22:00:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1253596160 bytes 2023-12-27T22:00:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1284457472 bytes 2023-12-27T22:00:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1321053184 bytes 2023-12-27T22:00:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1394960384 bytes 2023-12-27T22:00:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1464212480 bytes 2023-12-27T22:00:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1464843264 bytes 2023-12-27T22:00:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1465451520 bytes 2023-12-27T22:00:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1493966848 bytes 2023-12-27T22:00:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1548221440 bytes 2023-12-27T22:00:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1597670400 bytes 2023-12-27T22:00:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1662693376 bytes 2023-12-27T22:00:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1695032320 bytes 2023-12-27T22:00:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1695639552 bytes 2023-12-27T22:00:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1738560512 bytes 2023-12-27T22:00:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1767779328 bytes 2023-12-27T22:00:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1841634304 bytes 2023-12-27T22:00:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1983100928 bytes 2023-12-27T22:00:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2019236864 bytes 2023-12-27T22:00:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2055830528 bytes 2023-12-27T22:00:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2106605568 bytes 2023-12-27T22:00:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2157317120 bytes 2023-12-27T22:00:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2223968256 bytes 2023-12-27T22:00:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2282422272 bytes 2023-12-27T22:00:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2306705408 bytes 2023-12-27T22:00:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2436449280 bytes 2023-12-27T22:00:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2470934528 bytes 2023-12-27T22:00:19.091 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:00:19.090+0000 7f67e642e700 -1 reset not still connected to 0x55ca74ece0d0 2023-12-27T22:00:19.091 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:00:19.090+0000 7f67e642e700 -1 reset not still connected to 0x55ca766c80d0 2023-12-27T22:00:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2523962368 bytes 2023-12-27T22:00:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2528490496 bytes 2023-12-27T22:00:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2571978752 bytes 2023-12-27T22:00:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2572581888 bytes 2023-12-27T22:00:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2585149440 bytes 2023-12-27T22:00:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2660756480 bytes 2023-12-27T22:00:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2704571392 bytes 2023-12-27T22:00:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2725937152 bytes 2023-12-27T22:00:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2745570304 bytes 2023-12-27T22:00:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2777951232 bytes 2023-12-27T22:00:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2798905344 bytes 2023-12-27T22:00:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2840551424 bytes 2023-12-27T22:00:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2841158656 bytes 2023-12-27T22:00:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2883873792 bytes 2023-12-27T22:00:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2917598208 bytes 2023-12-27T22:00:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 2959074304 bytes 2023-12-27T22:00:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3029338112 bytes 2023-12-27T22:00:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3048308736 bytes 2023-12-27T22:00:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3101523968 bytes 2023-12-27T22:00:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3222560768 bytes 2023-12-27T22:00:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3244312576 bytes 2023-12-27T22:00:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3261434880 bytes 2023-12-27T22:00:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3311164416 bytes 2023-12-27T22:00:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3375600640 bytes 2023-12-27T22:00:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3413741568 bytes 2023-12-27T22:00:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3467401216 bytes 2023-12-27T22:00:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8100 IOs, 3492970496 bytes 2023-12-27T22:00:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:00:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:00:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:00:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:00:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:00:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_2 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-27T22:00:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:00:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-27T22:00:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:00:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-27T22:00:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:00:23.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-27T22:00:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_2': 2023-12-27T22:00:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:00:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:00:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:00:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 126835ac2e89 2023-12-27T22:00:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.126835ac2e89 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:23 2023 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:23 2023 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:23 2023 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: f4c094d4-f920-4b5f-ade9-72239f34d084 2023-12-27T22:00:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:00:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:00:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-27T22:00:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:00:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:00:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=10 2023-12-27T22:00:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:00:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:00:23.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 13017088 bytes 2023-12-27T22:00:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 92864512 bytes 2023-12-27T22:00:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 133074944 bytes 2023-12-27T22:00:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 186457088 bytes 2023-12-27T22:00:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 290222080 bytes 2023-12-27T22:00:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 312634368 bytes 2023-12-27T22:00:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 313255936 bytes 2023-12-27T22:00:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 365960192 bytes 2023-12-27T22:00:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 390125568 bytes 2023-12-27T22:00:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 500386816 bytes 2023-12-27T22:00:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 523999232 bytes 2023-12-27T22:00:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 605305856 bytes 2023-12-27T22:00:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 642232320 bytes 2023-12-27T22:00:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 656842752 bytes 2023-12-27T22:00:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 665701376 bytes 2023-12-27T22:00:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 728900608 bytes 2023-12-27T22:00:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 740934656 bytes 2023-12-27T22:00:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 802487296 bytes 2023-12-27T22:00:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 838547456 bytes 2023-12-27T22:00:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 889205760 bytes 2023-12-27T22:00:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 914276352 bytes 2023-12-27T22:00:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 960403456 bytes 2023-12-27T22:00:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 982189056 bytes 2023-12-27T22:00:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1085232128 bytes 2023-12-27T22:00:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1142511616 bytes 2023-12-27T22:00:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1178529792 bytes 2023-12-27T22:00:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1275887616 bytes 2023-12-27T22:00:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1283070976 bytes 2023-12-27T22:00:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1336164352 bytes 2023-12-27T22:00:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1379561472 bytes 2023-12-27T22:00:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1432943616 bytes 2023-12-27T22:00:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1473371136 bytes 2023-12-27T22:00:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1484806144 bytes 2023-12-27T22:00:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1548267520 bytes 2023-12-27T22:00:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1583202304 bytes 2023-12-27T22:00:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1651267584 bytes 2023-12-27T22:00:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1699999744 bytes 2023-12-27T22:00:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1727244288 bytes 2023-12-27T22:00:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1769467904 bytes 2023-12-27T22:00:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1822578688 bytes 2023-12-27T22:00:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1872574464 bytes 2023-12-27T22:00:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1977355264 bytes 2023-12-27T22:00:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1992849408 bytes 2023-12-27T22:00:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2056858624 bytes 2023-12-27T22:00:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2113543168 bytes 2023-12-27T22:00:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2165392384 bytes 2023-12-27T22:00:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2229399552 bytes 2023-12-27T22:00:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2261553152 bytes 2023-12-27T22:00:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2326224896 bytes 2023-12-27T22:00:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2352103424 bytes 2023-12-27T22:00:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2375920640 bytes 2023-12-27T22:00:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2376526848 bytes 2023-12-27T22:00:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2384117760 bytes 2023-12-27T22:00:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2406661120 bytes 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_3 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-27T22:00:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:00:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:00:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:00:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:00:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-27T22:00:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:00:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-27T22:00:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:00:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-27T22:00:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:00:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_3': 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 127456733997 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.127456733997 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:33 2023 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:33 2023 2023-12-27T22:00:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:33 2023 2023-12-27T22:00:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:00:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:00:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d21ceb08-f137-46c9-a16b-ae39e2b1220d 2023-12-27T22:00:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:00:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:00:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-27T22:00:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:00:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:00:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:00:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:00:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:00:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 57780224 bytes 2023-12-27T22:00:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 88331264 bytes 2023-12-27T22:00:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 151989248 bytes 2023-12-27T22:00:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 258605056 bytes 2023-12-27T22:00:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 281434112 bytes 2023-12-27T22:00:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 327836672 bytes 2023-12-27T22:00:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 370679808 bytes 2023-12-27T22:00:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 413362176 bytes 2023-12-27T22:00:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 521101312 bytes 2023-12-27T22:00:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 567139328 bytes 2023-12-27T22:00:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 649862144 bytes 2023-12-27T22:00:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 682678272 bytes 2023-12-27T22:00:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 709354496 bytes 2023-12-27T22:00:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 764770304 bytes 2023-12-27T22:00:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 784135168 bytes 2023-12-27T22:00:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 803653632 bytes 2023-12-27T22:00:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 812165120 bytes 2023-12-27T22:00:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 848628736 bytes 2023-12-27T22:00:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 915698688 bytes 2023-12-27T22:00:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 958700544 bytes 2023-12-27T22:00:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1028211712 bytes 2023-12-27T22:00:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1068322816 bytes 2023-12-27T22:00:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1109649408 bytes 2023-12-27T22:00:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1122497536 bytes 2023-12-27T22:00:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1128536064 bytes 2023-12-27T22:00:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1149477888 bytes 2023-12-27T22:00:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1209952256 bytes 2023-12-27T22:00:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1230808064 bytes 2023-12-27T22:00:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1286806528 bytes 2023-12-27T22:00:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1352841216 bytes 2023-12-27T22:00:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1437864960 bytes 2023-12-27T22:00:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1527974912 bytes 2023-12-27T22:00:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1571841024 bytes 2023-12-27T22:00:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1611878400 bytes 2023-12-27T22:00:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1660252160 bytes 2023-12-27T22:00:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1784351744 bytes 2023-12-27T22:00:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1830465536 bytes 2023-12-27T22:00:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1831088128 bytes 2023-12-27T22:00:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1864241152 bytes 2023-12-27T22:00:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1934243840 bytes 2023-12-27T22:00:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1985518592 bytes 2023-12-27T22:00:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 2123580416 bytes 2023-12-27T22:00:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 2202916864 bytes 2023-12-27T22:00:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2231824384 bytes 2023-12-27T22:00:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2267682816 bytes 2023-12-27T22:00:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2370139136 bytes 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_4 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-27T22:00:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:00:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:00:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:00:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:00:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-27T22:00:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:00:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-27T22:00:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:00:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-27T22:00:44.058 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T22:00:44.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T22:00:44.059 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T22:00:44.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:00:44.060+0000 7f497f350700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 53815) UID: 1000 2023-12-27T22:00:44.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:00:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:00:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-27T22:00:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_4': 2023-12-27T22:00:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:00:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:00:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:00:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1280f975e2bb 2023-12-27T22:00:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1280f975e2bb 2023-12-27T22:00:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:43 2023 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:43 2023 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:43 2023 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: c0d17036-eff0-4639-8b58-29f45ac3edc8 2023-12-27T22:00:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:00:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:00:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-27T22:00:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:00:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:00:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=8 2023-12-27T22:00:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:00:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:00:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 62002176 bytes 2023-12-27T22:00:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 118940672 bytes 2023-12-27T22:00:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 170494976 bytes 2023-12-27T22:00:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 216122368 bytes 2023-12-27T22:00:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 299624448 bytes 2023-12-27T22:00:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 416289792 bytes 2023-12-27T22:00:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 458973184 bytes 2023-12-27T22:00:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 509025280 bytes 2023-12-27T22:00:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 588120064 bytes 2023-12-27T22:00:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 602293248 bytes 2023-12-27T22:00:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 634126336 bytes 2023-12-27T22:00:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 669602816 bytes 2023-12-27T22:00:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 692376576 bytes 2023-12-27T22:00:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 741527552 bytes 2023-12-27T22:00:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 776628224 bytes 2023-12-27T22:00:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 777268224 bytes 2023-12-27T22:00:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 830462976 bytes 2023-12-27T22:00:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 862211072 bytes 2023-12-27T22:00:48.060 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T22:00:48.060 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:00:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:00:48.061 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T22:00:48.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T22:00:48.061 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T22:00:48.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T22:00:48.064 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2023-12-27T22:00:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 923034624 bytes 2023-12-27T22:00:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 992455680 bytes 2023-12-27T22:00:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1074480128 bytes 2023-12-27T22:00:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1141453824 bytes 2023-12-27T22:00:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1158504448 bytes 2023-12-27T22:00:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1221231616 bytes 2023-12-27T22:00:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1235909632 bytes 2023-12-27T22:00:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1328471040 bytes 2023-12-27T22:00:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1380178944 bytes 2023-12-27T22:00:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1406033920 bytes 2023-12-27T22:00:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1445164032 bytes 2023-12-27T22:00:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1499357184 bytes 2023-12-27T22:00:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1559734272 bytes 2023-12-27T22:00:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1593464832 bytes 2023-12-27T22:00:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1612697600 bytes 2023-12-27T22:00:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1628021760 bytes 2023-12-27T22:00:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1657551872 bytes 2023-12-27T22:00:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1719658496 bytes 2023-12-27T22:00:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1788015616 bytes 2023-12-27T22:00:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1847046144 bytes 2023-12-27T22:00:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1858254848 bytes 2023-12-27T22:00:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1903536128 bytes 2023-12-27T22:00:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 2002800640 bytes 2023-12-27T22:00:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 2048521216 bytes 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_5 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:00:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:00:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:00:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:00:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-27T22:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-27T22:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:00:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-27T22:00:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:00:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_5': 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1292116e9fc8 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1292116e9fc8 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:00:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:52 2023 2023-12-27T22:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:52 2023 2023-12-27T22:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:52 2023 2023-12-27T22:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d2ad7245-32f6-4c55-975e-2cc6979edf3e 2023-12-27T22:00:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-27T22:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:00:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:00:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T22:00:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:00:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:00:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 80523264 bytes 2023-12-27T22:00:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 112770048 bytes 2023-12-27T22:00:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 132557824 bytes 2023-12-27T22:00:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 155729920 bytes 2023-12-27T22:00:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 186466304 bytes 2023-12-27T22:00:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 241340416 bytes 2023-12-27T22:00:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 259895296 bytes 2023-12-27T22:00:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 265945088 bytes 2023-12-27T22:00:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 278528000 bytes 2023-12-27T22:00:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 312936448 bytes 2023-12-27T22:00:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 313548800 bytes 2023-12-27T22:00:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 323234816 bytes 2023-12-27T22:00:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 357528576 bytes 2023-12-27T22:00:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 395162624 bytes 2023-12-27T22:00:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 436458496 bytes 2023-12-27T22:00:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 464338944 bytes 2023-12-27T22:00:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 562621440 bytes 2023-12-27T22:00:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 584867840 bytes 2023-12-27T22:00:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 623683584 bytes 2023-12-27T22:00:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 662721536 bytes 2023-12-27T22:00:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 696136704 bytes 2023-12-27T22:00:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 730101760 bytes 2023-12-27T22:00:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 759212032 bytes 2023-12-27T22:00:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 786539520 bytes 2023-12-27T22:00:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 818387968 bytes 2023-12-27T22:00:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 854207488 bytes 2023-12-27T22:00:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 879144960 bytes 2023-12-27T22:00:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 897712128 bytes 2023-12-27T22:00:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 959762432 bytes 2023-12-27T22:00:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 997280768 bytes 2023-12-27T22:00:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1045519360 bytes 2023-12-27T22:00:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1115308032 bytes 2023-12-27T22:00:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1149303808 bytes 2023-12-27T22:00:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1186707456 bytes 2023-12-27T22:00:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1228053504 bytes 2023-12-27T22:00:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1295828992 bytes 2023-12-27T22:00:59.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1309133824 bytes 2023-12-27T22:00:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1338624000 bytes 2023-12-27T22:00:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1367270400 bytes 2023-12-27T22:00:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1373952000 bytes 2023-12-27T22:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1407254528 bytes 2023-12-27T22:01:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1453784064 bytes 2023-12-27T22:01:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1458648064 bytes 2023-12-27T22:01:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1474407424 bytes 2023-12-27T22:01:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1509513216 bytes 2023-12-27T22:01:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:01:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:01:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_6 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-27T22:01:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-27T22:01:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:01:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-27T22:01:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:01:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-27T22:01:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:01:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-27T22:01:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_6': 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 129ed82f85ce 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.129ed82f85ce 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:00 2023 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:00 2023 2023-12-27T22:01:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:00 2023 2023-12-27T22:01:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:01:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:01:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 7b6aaf8f-abec-4e77-a74c-774b6db907b0 2023-12-27T22:01:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:01:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:01:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-27T22:01:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:01:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:01:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=11 2023-12-27T22:01:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:01:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:01:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 33354752 bytes 2023-12-27T22:01:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 95345664 bytes 2023-12-27T22:01:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 104708096 bytes 2023-12-27T22:01:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 105323520 bytes 2023-12-27T22:01:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 123463680 bytes 2023-12-27T22:01:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 185699328 bytes 2023-12-27T22:01:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 186323968 bytes 2023-12-27T22:01:02.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 209842176 bytes 2023-12-27T22:01:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 247304192 bytes 2023-12-27T22:01:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 333506560 bytes 2023-12-27T22:01:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 413167616 bytes 2023-12-27T22:01:03.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 433513472 bytes 2023-12-27T22:01:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 449810432 bytes 2023-12-27T22:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 492545024 bytes 2023-12-27T22:01:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 529209344 bytes 2023-12-27T22:01:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 553072640 bytes 2023-12-27T22:01:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 622084096 bytes 2023-12-27T22:01:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 667338752 bytes 2023-12-27T22:01:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 684798976 bytes 2023-12-27T22:01:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 727745536 bytes 2023-12-27T22:01:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 733518848 bytes 2023-12-27T22:01:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 771903488 bytes 2023-12-27T22:01:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 819078144 bytes 2023-12-27T22:01:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 839051264 bytes 2023-12-27T22:01:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 924936192 bytes 2023-12-27T22:01:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 946846720 bytes 2023-12-27T22:01:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 991150080 bytes 2023-12-27T22:01:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1009027072 bytes 2023-12-27T22:01:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1037819904 bytes 2023-12-27T22:01:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1104766976 bytes 2023-12-27T22:01:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1150870528 bytes 2023-12-27T22:01:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1151499264 bytes 2023-12-27T22:01:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1234992128 bytes 2023-12-27T22:01:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1249489920 bytes 2023-12-27T22:01:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1361651712 bytes 2023-12-27T22:01:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1437700096 bytes 2023-12-27T22:01:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1478807552 bytes 2023-12-27T22:01:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1492274176 bytes 2023-12-27T22:01:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1526884352 bytes 2023-12-27T22:01:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1559573504 bytes 2023-12-27T22:01:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1621753856 bytes 2023-12-27T22:01:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1638711296 bytes 2023-12-27T22:01:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1654395904 bytes 2023-12-27T22:01:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1701628928 bytes 2023-12-27T22:01:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1713427456 bytes 2023-12-27T22:01:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1780616192 bytes 2023-12-27T22:01:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1801628672 bytes 2023-12-27T22:01:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1847111680 bytes 2023-12-27T22:01:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1878727680 bytes 2023-12-27T22:01:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1939471360 bytes 2023-12-27T22:01:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 1950432256 bytes 2023-12-27T22:01:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 1984076800 bytes 2023-12-27T22:01:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2049747968 bytes 2023-12-27T22:01:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2140131328 bytes 2023-12-27T22:01:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2166962176 bytes 2023-12-27T22:01:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2176273408 bytes 2023-12-27T22:01:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2198951936 bytes 2023-12-27T22:01:11.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2247151616 bytes 2023-12-27T22:01:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2257005568 bytes 2023-12-27T22:01:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2278260736 bytes 2023-12-27T22:01:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2407773184 bytes 2023-12-27T22:01:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2437755904 bytes 2023-12-27T22:01:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2462317568 bytes 2023-12-27T22:01:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2490215424 bytes 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_7 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:01:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:01:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-27T22:01:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:01:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-27T22:01:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:01:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-27T22:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:01:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-27T22:01:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_7': 2023-12-27T22:01:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:01:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12aa73e32d4a 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12aa73e32d4a 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:12 2023 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:12 2023 2023-12-27T22:01:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:12 2023 2023-12-27T22:01:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:01:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:01:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 5a81880e-ca19-4e3d-9afa-fac4d233817b 2023-12-27T22:01:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:01:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:01:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-27T22:01:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:01:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:01:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=13 2023-12-27T22:01:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:01:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:01:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 58046464 bytes 2023-12-27T22:01:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 128698368 bytes 2023-12-27T22:01:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 189463552 bytes 2023-12-27T22:01:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 267510784 bytes 2023-12-27T22:01:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 298308608 bytes 2023-12-27T22:01:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 321119232 bytes 2023-12-27T22:01:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 352376832 bytes 2023-12-27T22:01:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 469651456 bytes 2023-12-27T22:01:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 509583360 bytes 2023-12-27T22:01:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 520439808 bytes 2023-12-27T22:01:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 618964992 bytes 2023-12-27T22:01:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 701038592 bytes 2023-12-27T22:01:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 721116160 bytes 2023-12-27T22:01:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 725182464 bytes 2023-12-27T22:01:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 754982912 bytes 2023-12-27T22:01:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 813999104 bytes 2023-12-27T22:01:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 844954624 bytes 2023-12-27T22:01:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 845583360 bytes 2023-12-27T22:01:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 857930752 bytes 2023-12-27T22:01:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 883239936 bytes 2023-12-27T22:01:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 895583232 bytes 2023-12-27T22:01:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 923460608 bytes 2023-12-27T22:01:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 969479168 bytes 2023-12-27T22:01:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1012794368 bytes 2023-12-27T22:01:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1044291584 bytes 2023-12-27T22:01:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1064178688 bytes 2023-12-27T22:01:18.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3460465179448051) > weight (0.25) 2023-12-27T22:01:18.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7397189871168296) > weight (0.25) 2023-12-27T22:01:18.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7804803935864176) > weight (0.25) 2023-12-27T22:01:18.153 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4526446122704918) > weight (0.25) 2023-12-27T22:01:18.153 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-27T22:01:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1189151744 bytes 2023-12-27T22:01:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1261871104 bytes 2023-12-27T22:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1262483456 bytes 2023-12-27T22:01:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1285120000 bytes 2023-12-27T22:01:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1313786880 bytes 2023-12-27T22:01:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1340240896 bytes 2023-12-27T22:01:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1383418880 bytes 2023-12-27T22:01:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1384009728 bytes 2023-12-27T22:01:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1384623104 bytes 2023-12-27T22:01:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1445236736 bytes 2023-12-27T22:01:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1514613760 bytes 2023-12-27T22:01:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1562559488 bytes 2023-12-27T22:01:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1604377600 bytes 2023-12-27T22:01:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1611291648 bytes 2023-12-27T22:01:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1637060608 bytes 2023-12-27T22:01:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1642933248 bytes 2023-12-27T22:01:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1667496960 bytes 2023-12-27T22:01:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1685380096 bytes 2023-12-27T22:01:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1722548224 bytes 2023-12-27T22:01:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1770352640 bytes 2023-12-27T22:01:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1830652928 bytes 2023-12-27T22:01:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1864588288 bytes 2023-12-27T22:01:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1894265856 bytes 2023-12-27T22:01:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1930358784 bytes 2023-12-27T22:01:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2072637440 bytes 2023-12-27T22:01:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2121568256 bytes 2023-12-27T22:01:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2169787392 bytes 2023-12-27T22:01:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2251305984 bytes 2023-12-27T22:01:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2303408128 bytes 2023-12-27T22:01:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2328918016 bytes 2023-12-27T22:01:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2375055360 bytes 2023-12-27T22:01:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2440472576 bytes 2023-12-27T22:01:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2539334656 bytes 2023-12-27T22:01:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2549624832 bytes 2023-12-27T22:01:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2601375744 bytes 2023-12-27T22:01:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2664425472 bytes 2023-12-27T22:01:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2677564416 bytes 2023-12-27T22:01:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2704636928 bytes 2023-12-27T22:01:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2716420096 bytes 2023-12-27T22:01:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2723008512 bytes 2023-12-27T22:01:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2725101568 bytes 2023-12-27T22:01:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2786043904 bytes 2023-12-27T22:01:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2882900992 bytes 2023-12-27T22:01:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 2931683328 bytes 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_8 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-27T22:01:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:01:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:01:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:01:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:01:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-27T22:01:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:01:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-27T22:01:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:01:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-27T22:01:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:01:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_8': 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12b6f4b9a77b 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12b6f4b9a77b 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:26 2023 2023-12-27T22:01:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:26 2023 2023-12-27T22:01:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:26 2023 2023-12-27T22:01:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:01:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:01:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d4452c8f-ad61-428d-a4d6-8e781d2d9950 2023-12-27T22:01:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:01:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:01:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-27T22:01:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:01:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:01:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T22:01:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:01:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:01:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 31527936 bytes 2023-12-27T22:01:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 98167808 bytes 2023-12-27T22:01:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 111968256 bytes 2023-12-27T22:01:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 133653504 bytes 2023-12-27T22:01:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 164500480 bytes 2023-12-27T22:01:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 215967744 bytes 2023-12-27T22:01:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 275976192 bytes 2023-12-27T22:01:28.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 303046656 bytes 2023-12-27T22:01:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 360115200 bytes 2023-12-27T22:01:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 392728576 bytes 2023-12-27T22:01:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 502949888 bytes 2023-12-27T22:01:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 522449920 bytes 2023-12-27T22:01:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 586670080 bytes 2023-12-27T22:01:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 587309056 bytes 2023-12-27T22:01:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 604473344 bytes 2023-12-27T22:01:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 674575360 bytes 2023-12-27T22:01:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 690118656 bytes 2023-12-27T22:01:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 766572544 bytes 2023-12-27T22:01:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 818928640 bytes 2023-12-27T22:01:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 859549696 bytes 2023-12-27T22:01:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 907438080 bytes 2023-12-27T22:01:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 943950848 bytes 2023-12-27T22:01:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1008605184 bytes 2023-12-27T22:01:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1050503168 bytes 2023-12-27T22:01:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1094304768 bytes 2023-12-27T22:01:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1096292352 bytes 2023-12-27T22:01:32.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1155480576 bytes 2023-12-27T22:01:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1202967552 bytes 2023-12-27T22:01:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1210172416 bytes 2023-12-27T22:01:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1263741952 bytes 2023-12-27T22:01:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1295992832 bytes 2023-12-27T22:01:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1349677056 bytes 2023-12-27T22:01:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1395999744 bytes 2023-12-27T22:01:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1411443712 bytes 2023-12-27T22:01:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1488257024 bytes 2023-12-27T22:01:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1537790976 bytes 2023-12-27T22:01:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1583650816 bytes 2023-12-27T22:01:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1627836416 bytes 2023-12-27T22:01:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1685640192 bytes 2023-12-27T22:01:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1754714112 bytes 2023-12-27T22:01:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1807565824 bytes 2023-12-27T22:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1871616000 bytes 2023-12-27T22:01:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1941928960 bytes 2023-12-27T22:01:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2009738240 bytes 2023-12-27T22:01:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2055350272 bytes 2023-12-27T22:01:35.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2076101632 bytes 2023-12-27T22:01:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2113361920 bytes 2023-12-27T22:01:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2132160512 bytes 2023-12-27T22:01:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2154758144 bytes 2023-12-27T22:01:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2205647872 bytes 2023-12-27T22:01:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2284803072 bytes 2023-12-27T22:01:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2302009344 bytes 2023-12-27T22:01:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2347755520 bytes 2023-12-27T22:01:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2376418304 bytes 2023-12-27T22:01:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2409061376 bytes 2023-12-27T22:01:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2433746944 bytes 2023-12-27T22:01:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2461410304 bytes 2023-12-27T22:01:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2467769344 bytes 2023-12-27T22:01:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2495864832 bytes 2023-12-27T22:01:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2524609536 bytes 2023-12-27T22:01:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2642893824 bytes 2023-12-27T22:01:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2674611200 bytes 2023-12-27T22:01:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2695802880 bytes 2023-12-27T22:01:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2729171968 bytes 2023-12-27T22:01:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2762570752 bytes 2023-12-27T22:01:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2826959872 bytes 2023-12-27T22:01:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2871263232 bytes 2023-12-27T22:01:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2935503872 bytes 2023-12-27T22:01:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2995853312 bytes 2023-12-27T22:01:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 3045525504 bytes 2023-12-27T22:01:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3051843584 bytes 2023-12-27T22:01:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3069660160 bytes 2023-12-27T22:01:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3080351744 bytes 2023-12-27T22:01:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3107953664 bytes 2023-12-27T22:01:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3118430208 bytes 2023-12-27T22:01:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3156707328 bytes 2023-12-27T22:01:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3157560320 bytes 2023-12-27T22:01:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3191623680 bytes 2023-12-27T22:01:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:01:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:01:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_9 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-12-27T22:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-27T22:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-27T22:01:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:01:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-27T22:01:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_9': 2023-12-27T22:01:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12c2ffbf9db7 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12c2ffbf9db7 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:41 2023 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:41 2023 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:41 2023 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:01:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:01:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: bf528422-e47f-45b4-ada3-2e77c12c15d7 2023-12-27T22:01:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:01:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:01:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-27T22:01:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:01:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:01:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=12 2023-12-27T22:01:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:01:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:01:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 36296704 bytes 2023-12-27T22:01:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 81788928 bytes 2023-12-27T22:01:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 137718784 bytes 2023-12-27T22:01:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 213228544 bytes 2023-12-27T22:01:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 225801216 bytes 2023-12-27T22:01:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 251727872 bytes 2023-12-27T22:01:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 276195328 bytes 2023-12-27T22:01:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 290925568 bytes 2023-12-27T22:01:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 340080640 bytes 2023-12-27T22:01:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 415015936 bytes 2023-12-27T22:01:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 421719040 bytes 2023-12-27T22:01:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 458349568 bytes 2023-12-27T22:01:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 510903296 bytes 2023-12-27T22:01:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 546311168 bytes 2023-12-27T22:01:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 585310208 bytes 2023-12-27T22:01:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 631105536 bytes 2023-12-27T22:01:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 631702528 bytes 2023-12-27T22:01:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 689695744 bytes 2023-12-27T22:01:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 769332224 bytes 2023-12-27T22:01:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 834590720 bytes 2023-12-27T22:01:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 914472960 bytes 2023-12-27T22:01:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 932175872 bytes 2023-12-27T22:01:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 972018688 bytes 2023-12-27T22:01:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1002119168 bytes 2023-12-27T22:01:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1017865216 bytes 2023-12-27T22:01:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1034238976 bytes 2023-12-27T22:01:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1058397184 bytes 2023-12-27T22:01:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1125897216 bytes 2023-12-27T22:01:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1182369792 bytes 2023-12-27T22:01:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1257325568 bytes 2023-12-27T22:01:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1298245632 bytes 2023-12-27T22:01:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1409858560 bytes 2023-12-27T22:01:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1443913728 bytes 2023-12-27T22:01:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1557761024 bytes 2023-12-27T22:01:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1635972096 bytes 2023-12-27T22:01:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1664572416 bytes 2023-12-27T22:01:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1675204608 bytes 2023-12-27T22:01:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1731597312 bytes 2023-12-27T22:01:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1776946176 bytes 2023-12-27T22:01:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1840816128 bytes 2023-12-27T22:01:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1933444096 bytes 2023-12-27T22:01:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1971646464 bytes 2023-12-27T22:01:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 2003012608 bytes 2023-12-27T22:01:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2033972224 bytes 2023-12-27T22:01:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2065037312 bytes 2023-12-27T22:01:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2071126016 bytes 2023-12-27T22:01:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2079294464 bytes 2023-12-27T22:01:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2106498048 bytes 2023-12-27T22:01:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2178944000 bytes 2023-12-27T22:01:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2217569280 bytes 2023-12-27T22:01:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2223148032 bytes 2023-12-27T22:01:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2265559040 bytes 2023-12-27T22:01:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2316493824 bytes 2023-12-27T22:01:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2358953984 bytes 2023-12-27T22:01:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2405499904 bytes 2023-12-27T22:01:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2445800448 bytes 2023-12-27T22:01:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2509392896 bytes 2023-12-27T22:01:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2574132224 bytes 2023-12-27T22:01:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2623709184 bytes 2023-12-27T22:01:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2671986688 bytes 2023-12-27T22:01:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2725821440 bytes 2023-12-27T22:01:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2805084160 bytes 2023-12-27T22:01:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2831833088 bytes 2023-12-27T22:01:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2893406208 bytes 2023-12-27T22:01:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2919033856 bytes 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_10 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:01:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:01:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:01:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:01:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-12-27T22:01:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:01:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-27T22:01:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:01:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:01:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-27T22:01:55.064 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T22:01:55.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T22:01:55.065 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-27T22:01:55.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:01:55.067+0000 7f7fe8f1b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 56754) UID: 1000 2023-12-27T22:01:55.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:01:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:01:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-27T22:01:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_10': 2023-12-27T22:01:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:01:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:01:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:01:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12cedcc96148 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12cedcc96148 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:54 2023 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:54 2023 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:54 2023 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: ffe88c0d-d6ed-4f88-83b3-e604b69fe96e 2023-12-27T22:01:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:01:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:01:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-27T22:01:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:01:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:01:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:01:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:01:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=12 2023-12-27T22:01:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:01:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:01:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 26858496 bytes 2023-12-27T22:01:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 53245952 bytes 2023-12-27T22:01:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 53850112 bytes 2023-12-27T22:01:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 62307328 bytes 2023-12-27T22:01:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 95083520 bytes 2023-12-27T22:01:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 153923584 bytes 2023-12-27T22:01:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 163045376 bytes 2023-12-27T22:01:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 201647104 bytes 2023-12-27T22:01:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 216184832 bytes 2023-12-27T22:01:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 263424000 bytes 2023-12-27T22:01:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 280889344 bytes 2023-12-27T22:01:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 302763008 bytes 2023-12-27T22:01:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 326715392 bytes 2023-12-27T22:01:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 358562816 bytes 2023-12-27T22:01:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 408446976 bytes 2023-12-27T22:01:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 450543616 bytes 2023-12-27T22:01:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 540102656 bytes 2023-12-27T22:01:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 540703744 bytes 2023-12-27T22:01:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 580631552 bytes 2023-12-27T22:01:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 593021952 bytes 2023-12-27T22:01:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 608218112 bytes 2023-12-27T22:01:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 636342272 bytes 2023-12-27T22:01:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 639404032 bytes 2023-12-27T22:01:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 677816320 bytes 2023-12-27T22:01:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 727922688 bytes 2023-12-27T22:01:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 757743616 bytes 2023-12-27T22:02:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 834579456 bytes 2023-12-27T22:02:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 876608512 bytes 2023-12-27T22:02:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 950988800 bytes 2023-12-27T22:02:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 980942848 bytes 2023-12-27T22:02:01.066 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T22:02:01.066 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:02:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:02:01.066 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T22:02:01.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T22:02:01.067 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T22:02:01.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T22:02:01.070 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-27T22:02:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1005869056 bytes 2023-12-27T22:02:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1050506240 bytes 2023-12-27T22:02:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1105947648 bytes 2023-12-27T22:02:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1132308480 bytes 2023-12-27T22:02:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1132909568 bytes 2023-12-27T22:02:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1188128768 bytes 2023-12-27T22:02:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1243984896 bytes 2023-12-27T22:02:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1270712320 bytes 2023-12-27T22:02:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1290337280 bytes 2023-12-27T22:02:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1327310848 bytes 2023-12-27T22:02:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1351425024 bytes 2023-12-27T22:02:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1393132544 bytes 2023-12-27T22:02:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1439683584 bytes 2023-12-27T22:02:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1475684352 bytes 2023-12-27T22:02:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1528226816 bytes 2023-12-27T22:02:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1595963392 bytes 2023-12-27T22:02:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1616296960 bytes 2023-12-27T22:02:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1667224576 bytes 2023-12-27T22:02:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1774654464 bytes 2023-12-27T22:02:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1852111872 bytes 2023-12-27T22:02:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 1892211712 bytes 2023-12-27T22:02:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 1909785600 bytes 2023-12-27T22:02:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 1972684800 bytes 2023-12-27T22:02:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2032278528 bytes 2023-12-27T22:02:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2061722624 bytes 2023-12-27T22:02:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2112939008 bytes 2023-12-27T22:02:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2258338816 bytes 2023-12-27T22:02:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2285022208 bytes 2023-12-27T22:02:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2298519552 bytes 2023-12-27T22:02:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2392246272 bytes 2023-12-27T22:02:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2466942976 bytes 2023-12-27T22:02:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:02:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:02:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:02:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_11 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:02:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:02:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2023-12-27T22:02:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:02:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-27T22:02:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:02:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:07.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-27T22:02:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:02:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-27T22:02:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_11': 2023-12-27T22:02:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:02:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:02:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:02:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12e05bb9f292 2023-12-27T22:02:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12e05bb9f292 2023-12-27T22:02:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:07 2023 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:07 2023 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:07 2023 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 077dd2f0-f421-4104-9242-d6588dfb934d 2023-12-27T22:02:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:02:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:02:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-27T22:02:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:02:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:02:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T22:02:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:02:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:02:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 69238784 bytes 2023-12-27T22:02:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 156162048 bytes 2023-12-27T22:02:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 230747136 bytes 2023-12-27T22:02:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 289177600 bytes 2023-12-27T22:02:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 298442752 bytes 2023-12-27T22:02:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 313758720 bytes 2023-12-27T22:02:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 314372096 bytes 2023-12-27T22:02:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 355322880 bytes 2023-12-27T22:02:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 395422720 bytes 2023-12-27T22:02:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 412682240 bytes 2023-12-27T22:02:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 437997568 bytes 2023-12-27T22:02:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 479045632 bytes 2023-12-27T22:02:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 501046272 bytes 2023-12-27T22:02:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 536029184 bytes 2023-12-27T22:02:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 536674304 bytes 2023-12-27T22:02:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 560656384 bytes 2023-12-27T22:02:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 582520832 bytes 2023-12-27T22:02:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 650384384 bytes 2023-12-27T22:02:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 691326976 bytes 2023-12-27T22:02:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 707983360 bytes 2023-12-27T22:02:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 779770880 bytes 2023-12-27T22:02:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 804578304 bytes 2023-12-27T22:02:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 834308096 bytes 2023-12-27T22:02:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 917131264 bytes 2023-12-27T22:02:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 925907968 bytes 2023-12-27T22:02:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1004003328 bytes 2023-12-27T22:02:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1062722560 bytes 2023-12-27T22:02:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1133181952 bytes 2023-12-27T22:02:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1133787136 bytes 2023-12-27T22:02:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1148540928 bytes 2023-12-27T22:02:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1163394048 bytes 2023-12-27T22:02:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1254622208 bytes 2023-12-27T22:02:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1282472960 bytes 2023-12-27T22:02:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1324484608 bytes 2023-12-27T22:02:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1364990976 bytes 2023-12-27T22:02:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1420382208 bytes 2023-12-27T22:02:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1455476736 bytes 2023-12-27T22:02:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_12 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:02:15.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2023-12-27T22:02:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:02:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-27T22:02:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:02:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-27T22:02:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:02:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-27T22:02:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_12': 2023-12-27T22:02:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:02:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:02:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12eccd4fe8a 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12eccd4fe8a 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:15 2023 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:15 2023 2023-12-27T22:02:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:15 2023 2023-12-27T22:02:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:02:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:02:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: a041afaf-4f1b-4932-be77-439bad206770 2023-12-27T22:02:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:02:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:02:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-27T22:02:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:02:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:02:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=11 2023-12-27T22:02:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:02:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:02:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 20577280 bytes 2023-12-27T22:02:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 97545216 bytes 2023-12-27T22:02:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 138405888 bytes 2023-12-27T22:02:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 211024896 bytes 2023-12-27T22:02:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 217671680 bytes 2023-12-27T22:02:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 341984256 bytes 2023-12-27T22:02:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 378960896 bytes 2023-12-27T22:02:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 397412352 bytes 2023-12-27T22:02:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 419214336 bytes 2023-12-27T22:02:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 453410816 bytes 2023-12-27T22:02:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 559427584 bytes 2023-12-27T22:02:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 616025088 bytes 2023-12-27T22:02:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 645107712 bytes 2023-12-27T22:02:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 687152128 bytes 2023-12-27T22:02:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 730771456 bytes 2023-12-27T22:02:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 767756288 bytes 2023-12-27T22:02:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 785474560 bytes 2023-12-27T22:02:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 827364352 bytes 2023-12-27T22:02:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 870160384 bytes 2023-12-27T22:02:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 912951296 bytes 2023-12-27T22:02:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 913544192 bytes 2023-12-27T22:02:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 954505216 bytes 2023-12-27T22:02:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1015008256 bytes 2023-12-27T22:02:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1042257920 bytes 2023-12-27T22:02:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1060090880 bytes 2023-12-27T22:02:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1096701952 bytes 2023-12-27T22:02:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1148581888 bytes 2023-12-27T22:02:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1208032256 bytes 2023-12-27T22:02:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1279214592 bytes 2023-12-27T22:02:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1314332672 bytes 2023-12-27T22:02:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1381426176 bytes 2023-12-27T22:02:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1406891008 bytes 2023-12-27T22:02:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1461004288 bytes 2023-12-27T22:02:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1525028864 bytes 2023-12-27T22:02:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1592627200 bytes 2023-12-27T22:02:23.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1621060608 bytes 2023-12-27T22:02:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1640852480 bytes 2023-12-27T22:02:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1690250240 bytes 2023-12-27T22:02:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1755238400 bytes 2023-12-27T22:02:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1770486784 bytes 2023-12-27T22:02:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1833529344 bytes 2023-12-27T22:02:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1863767040 bytes 2023-12-27T22:02:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1911695360 bytes 2023-12-27T22:02:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1935772672 bytes 2023-12-27T22:02:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2003272704 bytes 2023-12-27T22:02:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2060113920 bytes 2023-12-27T22:02:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2111724544 bytes 2023-12-27T22:02:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2153972736 bytes 2023-12-27T22:02:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2225370112 bytes 2023-12-27T22:02:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2274749440 bytes 2023-12-27T22:02:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2322269184 bytes 2023-12-27T22:02:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2331237376 bytes 2023-12-27T22:02:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2381224960 bytes 2023-12-27T22:02:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2418259968 bytes 2023-12-27T22:02:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2458247168 bytes 2023-12-27T22:02:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2516186112 bytes 2023-12-27T22:02:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:02:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_13 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:02:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:02:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2023-12-27T22:02:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:02:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-27T22:02:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:02:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-27T22:02:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:02:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-27T22:02:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_13': 2023-12-27T22:02:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:02:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:02:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12f823dc2341 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12f823dc2341 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:27 2023 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:27 2023 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:27 2023 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: f7d5f611-7612-48c4-a55c-a9116d980d1b 2023-12-27T22:02:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:02:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:02:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-27T22:02:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:02:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:02:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T22:02:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:02:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:02:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 2529280 bytes 2023-12-27T22:02:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 49545216 bytes 2023-12-27T22:02:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 123046912 bytes 2023-12-27T22:02:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 144979968 bytes 2023-12-27T22:02:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 163172352 bytes 2023-12-27T22:02:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 212728832 bytes 2023-12-27T22:02:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 222368768 bytes 2023-12-27T22:02:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 262861824 bytes 2023-12-27T22:02:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 302243840 bytes 2023-12-27T22:02:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 396524544 bytes 2023-12-27T22:02:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 421207040 bytes 2023-12-27T22:02:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 437483520 bytes 2023-12-27T22:02:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 515665920 bytes 2023-12-27T22:02:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 584891392 bytes 2023-12-27T22:02:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 678907904 bytes 2023-12-27T22:02:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 707103744 bytes 2023-12-27T22:02:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 732413952 bytes 2023-12-27T22:02:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 797451264 bytes 2023-12-27T22:02:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 850042880 bytes 2023-12-27T22:02:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 889191424 bytes 2023-12-27T22:02:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 915820544 bytes 2023-12-27T22:02:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 937062400 bytes 2023-12-27T22:02:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 972726272 bytes 2023-12-27T22:02:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1029402624 bytes 2023-12-27T22:02:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1070761984 bytes 2023-12-27T22:02:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1087671296 bytes 2023-12-27T22:02:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1128009728 bytes 2023-12-27T22:02:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1195752448 bytes 2023-12-27T22:02:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1276541952 bytes 2023-12-27T22:02:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1291026432 bytes 2023-12-27T22:02:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1315565568 bytes 2023-12-27T22:02:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1324866560 bytes 2023-12-27T22:02:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1330287616 bytes 2023-12-27T22:02:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1352495104 bytes 2023-12-27T22:02:34.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1393711104 bytes 2023-12-27T22:02:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1434009600 bytes 2023-12-27T22:02:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1466440704 bytes 2023-12-27T22:02:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1495981056 bytes 2023-12-27T22:02:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1530014720 bytes 2023-12-27T22:02:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_14 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:02:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:02:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:02:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2023-12-27T22:02:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:02:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-27T22:02:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:02:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-27T22:02:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:02:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_14': 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1304d2e3f972 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1304d2e3f972 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:02:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:35 2023 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:35 2023 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:35 2023 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: cf062d67-4032-4099-9b25-373a590d58f7 2023-12-27T22:02:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:02:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:02:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-27T22:02:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:02:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:02:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T22:02:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:02:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:02:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 65366016 bytes 2023-12-27T22:02:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 100986880 bytes 2023-12-27T22:02:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 156661760 bytes 2023-12-27T22:02:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 251272192 bytes 2023-12-27T22:02:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 316542976 bytes 2023-12-27T22:02:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 354706432 bytes 2023-12-27T22:02:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 383216640 bytes 2023-12-27T22:02:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 383834112 bytes 2023-12-27T22:02:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 422645760 bytes 2023-12-27T22:02:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 478831616 bytes 2023-12-27T22:02:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 511429632 bytes 2023-12-27T22:02:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 582158336 bytes 2023-12-27T22:02:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 614896640 bytes 2023-12-27T22:02:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 650564608 bytes 2023-12-27T22:02:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 698339328 bytes 2023-12-27T22:02:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 783278080 bytes 2023-12-27T22:02:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 843932672 bytes 2023-12-27T22:02:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 878334976 bytes 2023-12-27T22:02:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 941811712 bytes 2023-12-27T22:02:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 981457920 bytes 2023-12-27T22:02:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1000970240 bytes 2023-12-27T22:02:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1018472448 bytes 2023-12-27T22:02:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1069273088 bytes 2023-12-27T22:02:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1127938048 bytes 2023-12-27T22:02:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1200227328 bytes 2023-12-27T22:02:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1235908608 bytes 2023-12-27T22:02:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1262681088 bytes 2023-12-27T22:02:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1265811456 bytes 2023-12-27T22:02:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1291190272 bytes 2023-12-27T22:02:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1321998336 bytes 2023-12-27T22:02:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1359505408 bytes 2023-12-27T22:02:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1403611136 bytes 2023-12-27T22:02:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1421693952 bytes 2023-12-27T22:02:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1487503360 bytes 2023-12-27T22:02:42.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1494073344 bytes 2023-12-27T22:02:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1562727424 bytes 2023-12-27T22:02:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1613981696 bytes 2023-12-27T22:02:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1657912320 bytes 2023-12-27T22:02:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1694664704 bytes 2023-12-27T22:02:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1728434176 bytes 2023-12-27T22:02:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1757551616 bytes 2023-12-27T22:02:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1805294592 bytes 2023-12-27T22:02:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1845188608 bytes 2023-12-27T22:02:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1875802112 bytes 2023-12-27T22:02:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1974232064 bytes 2023-12-27T22:02:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2090919936 bytes 2023-12-27T22:02:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2123445248 bytes 2023-12-27T22:02:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2167325696 bytes 2023-12-27T22:02:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2186039296 bytes 2023-12-27T22:02:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2246235136 bytes 2023-12-27T22:02:46.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2278509568 bytes 2023-12-27T22:02:46.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2302787584 bytes 2023-12-27T22:02:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2334661632 bytes 2023-12-27T22:02:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2349598720 bytes 2023-12-27T22:02:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2388740096 bytes 2023-12-27T22:02:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2402319360 bytes 2023-12-27T22:02:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2445262848 bytes 2023-12-27T22:02:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2553947136 bytes 2023-12-27T22:02:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2635408384 bytes 2023-12-27T22:02:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2692458496 bytes 2023-12-27T22:02:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2756274176 bytes 2023-12-27T22:02:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2786876416 bytes 2023-12-27T22:02:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2852860928 bytes 2023-12-27T22:02:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2881551360 bytes 2023-12-27T22:02:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2921276416 bytes 2023-12-27T22:02:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2935237632 bytes 2023-12-27T22:02:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 3041377280 bytes 2023-12-27T22:02:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 3061760000 bytes 2023-12-27T22:02:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 3098433536 bytes 2023-12-27T22:02:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 3197137920 bytes 2023-12-27T22:02:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3219338240 bytes 2023-12-27T22:02:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3307631616 bytes 2023-12-27T22:02:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3309822976 bytes 2023-12-27T22:02:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:02:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:02:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:02:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_15 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:02:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2023-12-27T22:02:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:02:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-27T22:02:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:02:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:02:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-27T22:02:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:02:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-27T22:02:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_15': 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 131056b8640c 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.131056b8640c 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:02:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:50 2023 2023-12-27T22:02:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:50 2023 2023-12-27T22:02:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:50 2023 2023-12-27T22:02:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:02:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:02:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 6b7b2978-4df8-43a6-826f-2914e643a2d2 2023-12-27T22:02:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:02:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:02:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-27T22:02:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:02:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:02:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:02:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:02:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=10 2023-12-27T22:02:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:02:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:02:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 25612288 bytes 2023-12-27T22:02:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 95950848 bytes 2023-12-27T22:02:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 96580608 bytes 2023-12-27T22:02:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 118572032 bytes 2023-12-27T22:02:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 134948864 bytes 2023-12-27T22:02:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 169841664 bytes 2023-12-27T22:02:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 194806784 bytes 2023-12-27T22:02:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 219500544 bytes 2023-12-27T22:02:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 245140480 bytes 2023-12-27T22:02:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 327329792 bytes 2023-12-27T22:02:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 344470528 bytes 2023-12-27T22:02:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 364178432 bytes 2023-12-27T22:02:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 395141120 bytes 2023-12-27T22:02:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 430876672 bytes 2023-12-27T22:02:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 483721216 bytes 2023-12-27T22:02:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 484341760 bytes 2023-12-27T22:02:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 522164224 bytes 2023-12-27T22:02:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 571363328 bytes 2023-12-27T22:02:54.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 643973120 bytes 2023-12-27T22:02:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 673998848 bytes 2023-12-27T22:02:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 698988544 bytes 2023-12-27T22:02:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 743149568 bytes 2023-12-27T22:02:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 780867584 bytes 2023-12-27T22:02:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 799842304 bytes 2023-12-27T22:02:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 845244416 bytes 2023-12-27T22:02:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 925160448 bytes 2023-12-27T22:02:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 973396992 bytes 2023-12-27T22:02:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1041855488 bytes 2023-12-27T22:02:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1073744896 bytes 2023-12-27T22:02:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1120023552 bytes 2023-12-27T22:02:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1141627904 bytes 2023-12-27T22:02:57.154 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-27T22:02:57.154 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-27T22:02:57.154 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-27T22:02:57.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T22:02:57.155+0000 7f692839b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 51006) UID: 1000 2023-12-27T22:02:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1160596480 bytes 2023-12-27T22:02:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1224725504 bytes 2023-12-27T22:02:57.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:02:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1255615488 bytes 2023-12-27T22:02:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1309358080 bytes 2023-12-27T22:02:58.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1402595328 bytes 2023-12-27T22:02:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1477071872 bytes 2023-12-27T22:02:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1542702080 bytes 2023-12-27T22:02:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1584915456 bytes 2023-12-27T22:02:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1610323968 bytes 2023-12-27T22:02:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1716815872 bytes 2023-12-27T22:02:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1730494464 bytes 2023-12-27T22:02:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1811117056 bytes 2023-12-27T22:02:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1841371136 bytes 2023-12-27T22:02:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1872326656 bytes 2023-12-27T22:03:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1930309632 bytes 2023-12-27T22:03:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1974187008 bytes 2023-12-27T22:03:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1974811648 bytes 2023-12-27T22:03:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2006233088 bytes 2023-12-27T22:03:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2026591232 bytes 2023-12-27T22:03:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2049676288 bytes 2023-12-27T22:03:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2116326400 bytes 2023-12-27T22:03:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2144498688 bytes 2023-12-27T22:03:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2271476736 bytes 2023-12-27T22:03:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_16 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:03:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2023-12-27T22:03:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:03:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-27T22:03:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:03:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-27T22:03:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:03:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_16': 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 131ba895f755 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.131ba895f755 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:03:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:03:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:01 2023 2023-12-27T22:03:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:01 2023 2023-12-27T22:03:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:01 2023 2023-12-27T22:03:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:03:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:03:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 299a19a7-69a7-45b4-804c-649ac34275ff 2023-12-27T22:03:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:03:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:03:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-27T22:03:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:03:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:03:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T22:03:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:03:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:03:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 54537216 bytes 2023-12-27T22:03:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 121872384 bytes 2023-12-27T22:03:03.155 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-27T22:03:03.155 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:03:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:03:03.156 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-27T22:03:03.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T22:03:03.156 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T22:03:03.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T22:03:03.159 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-27T22:03:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 194791424 bytes 2023-12-27T22:03:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 253111296 bytes 2023-12-27T22:03:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 334213120 bytes 2023-12-27T22:03:04.071 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T22:03:04.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T22:03:04.072 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-27T22:03:04.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:03:04.072+0000 7f7b3e987700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 57641) UID: 1000 2023-12-27T22:03:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 408097792 bytes 2023-12-27T22:03:04.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:03:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 467776512 bytes 2023-12-27T22:03:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 493997056 bytes 2023-12-27T22:03:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 520450048 bytes 2023-12-27T22:03:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 525493248 bytes 2023-12-27T22:03:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 563810304 bytes 2023-12-27T22:03:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 661456896 bytes 2023-12-27T22:03:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 698651648 bytes 2023-12-27T22:03:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 718153728 bytes 2023-12-27T22:03:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 784559104 bytes 2023-12-27T22:03:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 802370560 bytes 2023-12-27T22:03:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 895554560 bytes 2023-12-27T22:03:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 942882816 bytes 2023-12-27T22:03:06.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 969987072 bytes 2023-12-27T22:03:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1012674560 bytes 2023-12-27T22:03:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1032219648 bytes 2023-12-27T22:03:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1037435904 bytes 2023-12-27T22:03:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1095497728 bytes 2023-12-27T22:03:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1140695040 bytes 2023-12-27T22:03:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1176332288 bytes 2023-12-27T22:03:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1261586432 bytes 2023-12-27T22:03:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1307397120 bytes 2023-12-27T22:03:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1355642880 bytes 2023-12-27T22:03:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1393375232 bytes 2023-12-27T22:03:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1471033344 bytes 2023-12-27T22:03:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1472422912 bytes 2023-12-27T22:03:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1528357888 bytes 2023-12-27T22:03:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1550342144 bytes 2023-12-27T22:03:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1592858624 bytes 2023-12-27T22:03:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1609751552 bytes 2023-12-27T22:03:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1660219392 bytes 2023-12-27T22:03:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1710070784 bytes 2023-12-27T22:03:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1743519744 bytes 2023-12-27T22:03:10.072 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T22:03:10.072 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:03:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:03:10.073 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T22:03:10.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T22:03:10.073 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T22:03:10.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T22:03:10.077 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-27T22:03:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1773818880 bytes 2023-12-27T22:03:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1786316800 bytes 2023-12-27T22:03:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1874565120 bytes 2023-12-27T22:03:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1899393024 bytes 2023-12-27T22:03:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1970702336 bytes 2023-12-27T22:03:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2023565312 bytes 2023-12-27T22:03:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2063634432 bytes 2023-12-27T22:03:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2113512448 bytes 2023-12-27T22:03:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2139469824 bytes 2023-12-27T22:03:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2163756032 bytes 2023-12-27T22:03:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2164356096 bytes 2023-12-27T22:03:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2202227712 bytes 2023-12-27T22:03:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2237171712 bytes 2023-12-27T22:03:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2260638720 bytes 2023-12-27T22:03:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2293254144 bytes 2023-12-27T22:03:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2331749376 bytes 2023-12-27T22:03:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2407597056 bytes 2023-12-27T22:03:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2443635712 bytes 2023-12-27T22:03:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2463300608 bytes 2023-12-27T22:03:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2528932864 bytes 2023-12-27T22:03:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2550528000 bytes 2023-12-27T22:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2567975936 bytes 2023-12-27T22:03:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2644469760 bytes 2023-12-27T22:03:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2744381440 bytes 2023-12-27T22:03:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2769201152 bytes 2023-12-27T22:03:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2806404096 bytes 2023-12-27T22:03:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2873951232 bytes 2023-12-27T22:03:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2924394496 bytes 2023-12-27T22:03:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2953831424 bytes 2023-12-27T22:03:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2954677248 bytes 2023-12-27T22:03:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2981541888 bytes 2023-12-27T22:03:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 3020967936 bytes 2023-12-27T22:03:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3031237632 bytes 2023-12-27T22:03:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3085679616 bytes 2023-12-27T22:03:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3156829184 bytes 2023-12-27T22:03:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3181534208 bytes 2023-12-27T22:03:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3184409600 bytes 2023-12-27T22:03:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:03:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:03:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:03:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:03:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_17 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-27T22:03:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2023-12-27T22:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-27T22:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:16.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-27T22:03:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:03:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_17': 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13361dbe939a 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13361dbe939a 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:03:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:16 2023 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:16 2023 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:16 2023 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: fd00cd65-5352-4569-8fe7-a9ebb5daa8a3 2023-12-27T22:03:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:03:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:03:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-27T22:03:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:03:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:03:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=11 2023-12-27T22:03:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:03:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:03:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 45521920 bytes 2023-12-27T22:03:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 117236736 bytes 2023-12-27T22:03:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 147685376 bytes 2023-12-27T22:03:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 185657344 bytes 2023-12-27T22:03:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 246917120 bytes 2023-12-27T22:03:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 267073536 bytes 2023-12-27T22:03:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 306123776 bytes 2023-12-27T22:03:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 316601344 bytes 2023-12-27T22:03:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 364343296 bytes 2023-12-27T22:03:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 453170176 bytes 2023-12-27T22:03:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 453813248 bytes 2023-12-27T22:03:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 467575808 bytes 2023-12-27T22:03:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 516636672 bytes 2023-12-27T22:03:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 537773056 bytes 2023-12-27T22:03:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 581966848 bytes 2023-12-27T22:03:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 633597952 bytes 2023-12-27T22:03:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 668203008 bytes 2023-12-27T22:03:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 686052352 bytes 2023-12-27T22:03:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 717862912 bytes 2023-12-27T22:03:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 818311168 bytes 2023-12-27T22:03:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 878958592 bytes 2023-12-27T22:03:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 924489728 bytes 2023-12-27T22:03:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 925113344 bytes 2023-12-27T22:03:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 985400320 bytes 2023-12-27T22:03:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1086100480 bytes 2023-12-27T22:03:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1173490688 bytes 2023-12-27T22:03:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1222217728 bytes 2023-12-27T22:03:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1253614592 bytes 2023-12-27T22:03:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1272775680 bytes 2023-12-27T22:03:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1316188160 bytes 2023-12-27T22:03:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1316795392 bytes 2023-12-27T22:03:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1382726656 bytes 2023-12-27T22:03:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1422203904 bytes 2023-12-27T22:03:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1495231488 bytes 2023-12-27T22:03:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1565521920 bytes 2023-12-27T22:03:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1630922752 bytes 2023-12-27T22:03:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1741755392 bytes 2023-12-27T22:03:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1774657536 bytes 2023-12-27T22:03:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1797052416 bytes 2023-12-27T22:03:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1816927232 bytes 2023-12-27T22:03:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1886892032 bytes 2023-12-27T22:03:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1904777216 bytes 2023-12-27T22:03:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1933928448 bytes 2023-12-27T22:03:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1949539328 bytes 2023-12-27T22:03:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2055584768 bytes 2023-12-27T22:03:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2070513664 bytes 2023-12-27T22:03:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2090633216 bytes 2023-12-27T22:03:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2118632448 bytes 2023-12-27T22:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2135098368 bytes 2023-12-27T22:03:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2135717888 bytes 2023-12-27T22:03:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2201314304 bytes 2023-12-27T22:03:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2225592320 bytes 2023-12-27T22:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2296403968 bytes 2023-12-27T22:03:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2362545152 bytes 2023-12-27T22:03:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2407798784 bytes 2023-12-27T22:03:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2463203328 bytes 2023-12-27T22:03:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2476175360 bytes 2023-12-27T22:03:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2524247040 bytes 2023-12-27T22:03:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:03:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:03:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:03:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:03:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:03:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_18 2023-12-27T22:03:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:03:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:03:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:03:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:03:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2023-12-27T22:03:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:03:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-27T22:03:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:03:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-27T22:03:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:03:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_18': 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 134292c969b9 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.134292c969b9 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:03:29.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:28 2023 2023-12-27T22:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:28 2023 2023-12-27T22:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:28 2023 2023-12-27T22:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: e11e4f17-4382-4319-a42a-59efce51e962 2023-12-27T22:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:03:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-27T22:03:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:03:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:03:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=10 2023-12-27T22:03:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:03:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:03:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 615424 bytes 2023-12-27T22:03:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 20424704 bytes 2023-12-27T22:03:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 133005312 bytes 2023-12-27T22:03:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 216491008 bytes 2023-12-27T22:03:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 284516352 bytes 2023-12-27T22:03:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 329105408 bytes 2023-12-27T22:03:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 370280448 bytes 2023-12-27T22:03:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 379052032 bytes 2023-12-27T22:03:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 379661312 bytes 2023-12-27T22:03:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 387469312 bytes 2023-12-27T22:03:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 423442432 bytes 2023-12-27T22:03:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 467980288 bytes 2023-12-27T22:03:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 507731968 bytes 2023-12-27T22:03:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 557128704 bytes 2023-12-27T22:03:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 609355776 bytes 2023-12-27T22:03:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 643659776 bytes 2023-12-27T22:03:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 664247296 bytes 2023-12-27T22:03:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 706330624 bytes 2023-12-27T22:03:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 768141312 bytes 2023-12-27T22:03:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 798049280 bytes 2023-12-27T22:03:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 815539200 bytes 2023-12-27T22:03:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 898814976 bytes 2023-12-27T22:03:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 943053824 bytes 2023-12-27T22:03:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 964972544 bytes 2023-12-27T22:03:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1007270912 bytes 2023-12-27T22:03:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1037143040 bytes 2023-12-27T22:03:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1076841472 bytes 2023-12-27T22:03:34.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1092494336 bytes 2023-12-27T22:03:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1152818176 bytes 2023-12-27T22:03:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1233600512 bytes 2023-12-27T22:03:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1337802752 bytes 2023-12-27T22:03:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1343862784 bytes 2023-12-27T22:03:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1396379648 bytes 2023-12-27T22:03:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1436066816 bytes 2023-12-27T22:03:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1482814464 bytes 2023-12-27T22:03:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1500148736 bytes 2023-12-27T22:03:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1543729152 bytes 2023-12-27T22:03:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1550300160 bytes 2023-12-27T22:03:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1576000512 bytes 2023-12-27T22:03:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1616204800 bytes 2023-12-27T22:03:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1668498432 bytes 2023-12-27T22:03:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1709694976 bytes 2023-12-27T22:03:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1737409536 bytes 2023-12-27T22:03:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1777215488 bytes 2023-12-27T22:03:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1814974464 bytes 2023-12-27T22:03:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1903121408 bytes 2023-12-27T22:03:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1945199616 bytes 2023-12-27T22:03:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1969649664 bytes 2023-12-27T22:03:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1990701056 bytes 2023-12-27T22:03:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2029528064 bytes 2023-12-27T22:03:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2085608448 bytes 2023-12-27T22:03:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2244156416 bytes 2023-12-27T22:03:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:03:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:03:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_19 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:03:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:03:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-27T22:03:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:03:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:03:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:03:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:03:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2023-12-27T22:03:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:03:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-27T22:03:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:03:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-27T22:03:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:03:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-27T22:03:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_19': 2023-12-27T22:03:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 134e996786f0 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.134e996786f0 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:39 2023 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:39 2023 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:39 2023 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: f3df9bf1-fabf-4121-bdef-2b2d3451c060 2023-12-27T22:03:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:03:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:03:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-27T22:03:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:03:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:03:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:03:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:03:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:03:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 31723520 bytes 2023-12-27T22:03:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 43734016 bytes 2023-12-27T22:03:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 105337856 bytes 2023-12-27T22:03:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 139517952 bytes 2023-12-27T22:03:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 182205440 bytes 2023-12-27T22:03:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 202492928 bytes 2023-12-27T22:03:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 240321536 bytes 2023-12-27T22:03:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 282013696 bytes 2023-12-27T22:03:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 327220224 bytes 2023-12-27T22:03:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 365654016 bytes 2023-12-27T22:03:42.659 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:03:42.660+0000 7fa0daabc700 -1 reset not still connected to 0x5566fe6a8b60 2023-12-27T22:03:42.660 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:03:42.660+0000 7fa0daabc700 -1 reset not still connected to 0x55670f6164e0 2023-12-27T22:03:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 421316608 bytes 2023-12-27T22:03:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 455640064 bytes 2023-12-27T22:03:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 505362432 bytes 2023-12-27T22:03:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 505989120 bytes 2023-12-27T22:03:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 576154624 bytes 2023-12-27T22:03:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 594493440 bytes 2023-12-27T22:03:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 648069120 bytes 2023-12-27T22:03:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 706960384 bytes 2023-12-27T22:03:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 753292288 bytes 2023-12-27T22:03:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 816766976 bytes 2023-12-27T22:03:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 836284416 bytes 2023-12-27T22:03:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 836910080 bytes 2023-12-27T22:03:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 886729728 bytes 2023-12-27T22:03:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 939246592 bytes 2023-12-27T22:03:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 987405312 bytes 2023-12-27T22:03:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1013336064 bytes 2023-12-27T22:03:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1079855104 bytes 2023-12-27T22:03:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1104788480 bytes 2023-12-27T22:03:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1138606080 bytes 2023-12-27T22:03:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1196514304 bytes 2023-12-27T22:03:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1297641472 bytes 2023-12-27T22:03:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1392605184 bytes 2023-12-27T22:03:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1469744128 bytes 2023-12-27T22:03:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1515399168 bytes 2023-12-27T22:03:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1585803264 bytes 2023-12-27T22:03:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1609745408 bytes 2023-12-27T22:03:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1670149120 bytes 2023-12-27T22:03:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1721910272 bytes 2023-12-27T22:03:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1774298112 bytes 2023-12-27T22:03:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1805517824 bytes 2023-12-27T22:03:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1862608896 bytes 2023-12-27T22:03:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1905347584 bytes 2023-12-27T22:03:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1949961216 bytes 2023-12-27T22:03:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1978095616 bytes 2023-12-27T22:03:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2051860480 bytes 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_20 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:03:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:03:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:03:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:03:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:03:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2023-12-27T22:03:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:03:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-27T22:03:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:03:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:03:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-27T22:03:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:03:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-27T22:03:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_20': 2023-12-27T22:03:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:03:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:03:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 135a77f1b6ea 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.135a77f1b6ea 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:49 2023 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:49 2023 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:49 2023 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 7149049a-7770-48be-805f-bc4a325f8c4b 2023-12-27T22:03:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:03:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:03:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-27T22:03:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:03:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:03:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:03:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:03:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=11 2023-12-27T22:03:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:03:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:03:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 90708992 bytes 2023-12-27T22:03:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 166220800 bytes 2023-12-27T22:03:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 166824960 bytes 2023-12-27T22:03:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 227441664 bytes 2023-12-27T22:03:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 274914304 bytes 2023-12-27T22:03:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 286324736 bytes 2023-12-27T22:03:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 362946560 bytes 2023-12-27T22:03:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 380654592 bytes 2023-12-27T22:03:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 447602688 bytes 2023-12-27T22:03:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 500652032 bytes 2023-12-27T22:03:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 598034432 bytes 2023-12-27T22:03:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 626158592 bytes 2023-12-27T22:03:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 628317184 bytes 2023-12-27T22:03:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 695452672 bytes 2023-12-27T22:03:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 761672704 bytes 2023-12-27T22:03:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 800668672 bytes 2023-12-27T22:03:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 839516160 bytes 2023-12-27T22:03:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 899683328 bytes 2023-12-27T22:03:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 982760448 bytes 2023-12-27T22:03:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1030303744 bytes 2023-12-27T22:03:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1112115200 bytes 2023-12-27T22:03:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1127627776 bytes 2023-12-27T22:03:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1143396352 bytes 2023-12-27T22:03:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1211120640 bytes 2023-12-27T22:03:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1270800384 bytes 2023-12-27T22:03:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1317211136 bytes 2023-12-27T22:03:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1374348288 bytes 2023-12-27T22:03:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1397829632 bytes 2023-12-27T22:03:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1416892416 bytes 2023-12-27T22:03:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1463181312 bytes 2023-12-27T22:03:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1542571008 bytes 2023-12-27T22:03:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1581620224 bytes 2023-12-27T22:03:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1628289024 bytes 2023-12-27T22:03:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1682843648 bytes 2023-12-27T22:03:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1692934144 bytes 2023-12-27T22:03:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1722722304 bytes 2023-12-27T22:03:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1753367552 bytes 2023-12-27T22:03:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1811089408 bytes 2023-12-27T22:03:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1824944128 bytes 2023-12-27T22:03:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1883770880 bytes 2023-12-27T22:03:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1915814912 bytes 2023-12-27T22:03:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1938985984 bytes 2023-12-27T22:03:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 2006822912 bytes 2023-12-27T22:03:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2031044608 bytes 2023-12-27T22:03:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2033631232 bytes 2023-12-27T22:03:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2044786688 bytes 2023-12-27T22:03:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2060535808 bytes 2023-12-27T22:03:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2096841728 bytes 2023-12-27T22:04:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2099941376 bytes 2023-12-27T22:04:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2110651392 bytes 2023-12-27T22:04:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2142457856 bytes 2023-12-27T22:04:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2162578432 bytes 2023-12-27T22:04:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2248610816 bytes 2023-12-27T22:04:01.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2319234048 bytes 2023-12-27T22:04:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2343742464 bytes 2023-12-27T22:04:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2413206528 bytes 2023-12-27T22:04:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2446475264 bytes 2023-12-27T22:04:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:04:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:04:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:04:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:04:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_21 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2023-12-27T22:04:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:04:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-27T22:04:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:04:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-27T22:04:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:04:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-27T22:04:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_21': 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1366f8ba036a 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1366f8ba036a 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:01 2023 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:01 2023 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:01 2023 2023-12-27T22:04:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:04:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:04:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 06d7ea5e-0a1f-42f1-8ff9-fb3499537169 2023-12-27T22:04:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:04:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:04:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-27T22:04:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:04:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:04:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=10 2023-12-27T22:04:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:04:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:04:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 19355648 bytes 2023-12-27T22:04:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 40868864 bytes 2023-12-27T22:04:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 85265408 bytes 2023-12-27T22:04:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 119508992 bytes 2023-12-27T22:04:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 203567104 bytes 2023-12-27T22:04:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 250012672 bytes 2023-12-27T22:04:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 327023616 bytes 2023-12-27T22:04:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 429270016 bytes 2023-12-27T22:04:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 452518912 bytes 2023-12-27T22:04:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 525326336 bytes 2023-12-27T22:04:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 563271680 bytes 2023-12-27T22:04:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 624678912 bytes 2023-12-27T22:04:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 653581312 bytes 2023-12-27T22:04:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 667815936 bytes 2023-12-27T22:04:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 754611200 bytes 2023-12-27T22:04:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 836073472 bytes 2023-12-27T22:04:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 874419200 bytes 2023-12-27T22:04:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 924900352 bytes 2023-12-27T22:04:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 925514752 bytes 2023-12-27T22:04:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1020739584 bytes 2023-12-27T22:04:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1025599488 bytes 2023-12-27T22:04:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1052310528 bytes 2023-12-27T22:04:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1096322048 bytes 2023-12-27T22:04:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1096938496 bytes 2023-12-27T22:04:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1136630784 bytes 2023-12-27T22:04:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1186677760 bytes 2023-12-27T22:04:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1219753984 bytes 2023-12-27T22:04:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1311875072 bytes 2023-12-27T22:04:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1370224640 bytes 2023-12-27T22:04:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1451435008 bytes 2023-12-27T22:04:09.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1484716032 bytes 2023-12-27T22:04:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1531566080 bytes 2023-12-27T22:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1563651072 bytes 2023-12-27T22:04:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1586384896 bytes 2023-12-27T22:04:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1623066624 bytes 2023-12-27T22:04:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1671143424 bytes 2023-12-27T22:04:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1724756992 bytes 2023-12-27T22:04:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1764149248 bytes 2023-12-27T22:04:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1821583360 bytes 2023-12-27T22:04:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1848159232 bytes 2023-12-27T22:04:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1910497280 bytes 2023-12-27T22:04:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1947038720 bytes 2023-12-27T22:04:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1961809920 bytes 2023-12-27T22:04:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1965546496 bytes 2023-12-27T22:04:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1996441600 bytes 2023-12-27T22:04:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2049864704 bytes 2023-12-27T22:04:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2082462720 bytes 2023-12-27T22:04:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2186649600 bytes 2023-12-27T22:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2207500288 bytes 2023-12-27T22:04:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2280002560 bytes 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_22 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:04:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:04:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:04:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:04:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2023-12-27T22:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-27T22:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-27T22:04:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:04:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-27T22:04:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_22': 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13727ea2224a 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13727ea2224a 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:12 2023 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:12 2023 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:12 2023 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:04:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:04:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 18ab07a2-986f-4749-9454-ce1d87f8014f 2023-12-27T22:04:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:04:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:04:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-27T22:04:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:04:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:04:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=12 2023-12-27T22:04:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:04:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:04:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 33676288 bytes 2023-12-27T22:04:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 91461632 bytes 2023-12-27T22:04:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 161883136 bytes 2023-12-27T22:04:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 267552768 bytes 2023-12-27T22:04:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 303509504 bytes 2023-12-27T22:04:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 318733312 bytes 2023-12-27T22:04:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 376815616 bytes 2023-12-27T22:04:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 437959680 bytes 2023-12-27T22:04:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 509295616 bytes 2023-12-27T22:04:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 540946432 bytes 2023-12-27T22:04:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 585559040 bytes 2023-12-27T22:04:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 609037312 bytes 2023-12-27T22:04:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 672452608 bytes 2023-12-27T22:04:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 691894272 bytes 2023-12-27T22:04:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 699198464 bytes 2023-12-27T22:04:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 778144768 bytes 2023-12-27T22:04:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 838423552 bytes 2023-12-27T22:04:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 869968896 bytes 2023-12-27T22:04:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 914102272 bytes 2023-12-27T22:04:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 928877568 bytes 2023-12-27T22:04:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 954472448 bytes 2023-12-27T22:04:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 993212416 bytes 2023-12-27T22:04:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1026287616 bytes 2023-12-27T22:04:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1107099648 bytes 2023-12-27T22:04:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1111013376 bytes 2023-12-27T22:04:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1168316416 bytes 2023-12-27T22:04:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1178829824 bytes 2023-12-27T22:04:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1221588992 bytes 2023-12-27T22:04:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1233797120 bytes 2023-12-27T22:04:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1258361856 bytes 2023-12-27T22:04:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1291693056 bytes 2023-12-27T22:04:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1321042944 bytes 2023-12-27T22:04:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1350237184 bytes 2023-12-27T22:04:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1389800448 bytes 2023-12-27T22:04:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1413084160 bytes 2023-12-27T22:04:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1471748096 bytes 2023-12-27T22:04:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1505684480 bytes 2023-12-27T22:04:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1521292288 bytes 2023-12-27T22:04:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1533905920 bytes 2023-12-27T22:04:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1534496768 bytes 2023-12-27T22:04:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1583431680 bytes 2023-12-27T22:04:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1617777664 bytes 2023-12-27T22:04:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1706379264 bytes 2023-12-27T22:04:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1721262080 bytes 2023-12-27T22:04:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1752735744 bytes 2023-12-27T22:04:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1769540608 bytes 2023-12-27T22:04:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1813137408 bytes 2023-12-27T22:04:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1813761024 bytes 2023-12-27T22:04:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1835678720 bytes 2023-12-27T22:04:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1886438400 bytes 2023-12-27T22:04:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 1952491520 bytes 2023-12-27T22:04:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2004700160 bytes 2023-12-27T22:04:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2039168000 bytes 2023-12-27T22:04:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2089577472 bytes 2023-12-27T22:04:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2144975872 bytes 2023-12-27T22:04:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2168651776 bytes 2023-12-27T22:04:24.078 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7989036498972575) > weight (0.25) 2023-12-27T22:04:24.078 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6278777579402405) > weight (0.25) 2023-12-27T22:04:24.078 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7570638524538253) > weight (0.25) 2023-12-27T22:04:24.078 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4772331068972687) > weight (0.25) 2023-12-27T22:04:24.078 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-27T22:04:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2205147136 bytes 2023-12-27T22:04:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2212408320 bytes 2023-12-27T22:04:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2301083648 bytes 2023-12-27T22:04:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2355621888 bytes 2023-12-27T22:04:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2387739648 bytes 2023-12-27T22:04:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2433080320 bytes 2023-12-27T22:04:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2479704064 bytes 2023-12-27T22:04:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2516268032 bytes 2023-12-27T22:04:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:04:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:04:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:04:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:04:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:04:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_23 2023-12-27T22:04:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:04:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:04:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:04:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:04:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2023-12-27T22:04:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:04:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-27T22:04:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:04:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-27T22:04:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:04:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-27T22:04:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_23': 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 137e8d80c821 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.137e8d80c821 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:25 2023 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:25 2023 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:25 2023 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:04:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:04:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: a37ade65-cdd8-4035-b15e-da4bd79e5619 2023-12-27T22:04:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:04:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:04:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-27T22:04:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:04:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:04:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T22:04:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:04:26.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:04:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 59787264 bytes 2023-12-27T22:04:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 86279168 bytes 2023-12-27T22:04:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 146584576 bytes 2023-12-27T22:04:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 147187712 bytes 2023-12-27T22:04:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 147786752 bytes 2023-12-27T22:04:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 220865536 bytes 2023-12-27T22:04:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 245152768 bytes 2023-12-27T22:04:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 254587904 bytes 2023-12-27T22:04:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 290193408 bytes 2023-12-27T22:04:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 330919936 bytes 2023-12-27T22:04:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 342586368 bytes 2023-12-27T22:04:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 364875776 bytes 2023-12-27T22:04:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 412286976 bytes 2023-12-27T22:04:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 480706560 bytes 2023-12-27T22:04:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 527131648 bytes 2023-12-27T22:04:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 563010560 bytes 2023-12-27T22:04:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 606308352 bytes 2023-12-27T22:04:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 610710528 bytes 2023-12-27T22:04:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 625884160 bytes 2023-12-27T22:04:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 692792320 bytes 2023-12-27T22:04:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 734223360 bytes 2023-12-27T22:04:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 777122816 bytes 2023-12-27T22:04:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 820098048 bytes 2023-12-27T22:04:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 860418048 bytes 2023-12-27T22:04:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 886793216 bytes 2023-12-27T22:04:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 937126912 bytes 2023-12-27T22:04:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1055481856 bytes 2023-12-27T22:04:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1109870592 bytes 2023-12-27T22:04:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1138049024 bytes 2023-12-27T22:04:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1154050048 bytes 2023-12-27T22:04:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1199187968 bytes 2023-12-27T22:04:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1274530816 bytes 2023-12-27T22:04:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:04:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:04:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:04:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:04:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:04:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_24 2023-12-27T22:04:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2023-12-27T22:04:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:04:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-27T22:04:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:04:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-27T22:04:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:04:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_24': 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 138a85d96d0b 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.138a85d96d0b 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:32 2023 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:32 2023 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:32 2023 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 77a94122-1c65-4804-8b45-6573f18ab770 2023-12-27T22:04:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-27T22:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:04:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:04:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=13 2023-12-27T22:04:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:04:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:04:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 83661824 bytes 2023-12-27T22:04:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 128677888 bytes 2023-12-27T22:04:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 156550144 bytes 2023-12-27T22:04:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 220366848 bytes 2023-12-27T22:04:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 288199680 bytes 2023-12-27T22:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 340241408 bytes 2023-12-27T22:04:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 394416128 bytes 2023-12-27T22:04:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 414352384 bytes 2023-12-27T22:04:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 414967808 bytes 2023-12-27T22:04:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 455584768 bytes 2023-12-27T22:04:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 465760256 bytes 2023-12-27T22:04:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 510397440 bytes 2023-12-27T22:04:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 546118656 bytes 2023-12-27T22:04:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 615242752 bytes 2023-12-27T22:04:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 631116800 bytes 2023-12-27T22:04:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 710991872 bytes 2023-12-27T22:04:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 848289792 bytes 2023-12-27T22:04:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 871608320 bytes 2023-12-27T22:04:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 913310720 bytes 2023-12-27T22:04:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 981458944 bytes 2023-12-27T22:04:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1049613312 bytes 2023-12-27T22:04:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1103932416 bytes 2023-12-27T22:04:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1141708800 bytes 2023-12-27T22:04:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1203982336 bytes 2023-12-27T22:04:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1248479232 bytes 2023-12-27T22:04:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1296011264 bytes 2023-12-27T22:04:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1331875840 bytes 2023-12-27T22:04:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1391374336 bytes 2023-12-27T22:04:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1471330304 bytes 2023-12-27T22:04:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1520690176 bytes 2023-12-27T22:04:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1544987648 bytes 2023-12-27T22:04:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1564301312 bytes 2023-12-27T22:04:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1588793344 bytes 2023-12-27T22:04:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1596812288 bytes 2023-12-27T22:04:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1632451584 bytes 2023-12-27T22:04:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1672443904 bytes 2023-12-27T22:04:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1685727232 bytes 2023-12-27T22:04:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1767700480 bytes 2023-12-27T22:04:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1790814208 bytes 2023-12-27T22:04:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1800328192 bytes 2023-12-27T22:04:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1825316864 bytes 2023-12-27T22:04:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1844881408 bytes 2023-12-27T22:04:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1888220160 bytes 2023-12-27T22:04:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1967475712 bytes 2023-12-27T22:04:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2016467968 bytes 2023-12-27T22:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2035840000 bytes 2023-12-27T22:04:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2040218624 bytes 2023-12-27T22:04:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2053625856 bytes 2023-12-27T22:04:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2119451648 bytes 2023-12-27T22:04:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2185205760 bytes 2023-12-27T22:04:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2218145792 bytes 2023-12-27T22:04:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2265973760 bytes 2023-12-27T22:04:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2320694272 bytes 2023-12-27T22:04:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2344052736 bytes 2023-12-27T22:04:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2394064896 bytes 2023-12-27T22:04:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2443089920 bytes 2023-12-27T22:04:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2499915776 bytes 2023-12-27T22:04:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2594753536 bytes 2023-12-27T22:04:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2603283456 bytes 2023-12-27T22:04:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2632474624 bytes 2023-12-27T22:04:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2673524736 bytes 2023-12-27T22:04:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2721842176 bytes 2023-12-27T22:04:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2793454592 bytes 2023-12-27T22:04:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2845385728 bytes 2023-12-27T22:04:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2875125760 bytes 2023-12-27T22:04:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2919797760 bytes 2023-12-27T22:04:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_25 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:04:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2023-12-27T22:04:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:04:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-27T22:04:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:04:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-27T22:04:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:04:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-27T22:04:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_25': 2023-12-27T22:04:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:04:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1396b95170df 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1396b95170df 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:46 2023 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:46 2023 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:46 2023 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: ffe3b98c-986e-4f1e-bd91-bea8441372c0 2023-12-27T22:04:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:04:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:04:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-27T22:04:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:04:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:04:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=8 2023-12-27T22:04:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:04:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:04:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 54212608 bytes 2023-12-27T22:04:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 91724800 bytes 2023-12-27T22:04:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 137553920 bytes 2023-12-27T22:04:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 191370240 bytes 2023-12-27T22:04:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 249909248 bytes 2023-12-27T22:04:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 281000960 bytes 2023-12-27T22:04:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 348611584 bytes 2023-12-27T22:04:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 415730688 bytes 2023-12-27T22:04:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 541471744 bytes 2023-12-27T22:04:49.159 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-27T22:04:49.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-27T22:04:49.160 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-27T22:04:49.161 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T22:04:49.161+0000 7f8d5c788700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 58278) UID: 1000 2023-12-27T22:04:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 629839872 bytes 2023-12-27T22:04:49.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:04:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 650775552 bytes 2023-12-27T22:04:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 676589568 bytes 2023-12-27T22:04:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 753479680 bytes 2023-12-27T22:04:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 765915136 bytes 2023-12-27T22:04:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 768509952 bytes 2023-12-27T22:04:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 827243520 bytes 2023-12-27T22:04:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 868742144 bytes 2023-12-27T22:04:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 871630848 bytes 2023-12-27T22:04:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 873015296 bytes 2023-12-27T22:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 923025408 bytes 2023-12-27T22:04:51.161 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-27T22:04:51.161 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:04:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:04:51.162 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-27T22:04:51.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T22:04:51.162 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T22:04:51.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T22:04:51.166 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-27T22:04:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1010848768 bytes 2023-12-27T22:04:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1028020224 bytes 2023-12-27T22:04:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1091505152 bytes 2023-12-27T22:04:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1154708480 bytes 2023-12-27T22:04:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1220267008 bytes 2023-12-27T22:04:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1234770944 bytes 2023-12-27T22:04:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1268139008 bytes 2023-12-27T22:04:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1291614208 bytes 2023-12-27T22:04:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1342830592 bytes 2023-12-27T22:04:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1369630720 bytes 2023-12-27T22:04:53.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1424044032 bytes 2023-12-27T22:04:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1443515392 bytes 2023-12-27T22:04:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1459665920 bytes 2023-12-27T22:04:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1519784960 bytes 2023-12-27T22:04:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1566460928 bytes 2023-12-27T22:04:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1572957184 bytes 2023-12-27T22:04:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1593566208 bytes 2023-12-27T22:04:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1639668736 bytes 2023-12-27T22:04:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1687368704 bytes 2023-12-27T22:04:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1708933120 bytes 2023-12-27T22:04:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1739994112 bytes 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_26 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-27T22:04:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:04:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:04:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:04:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:04:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:04:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2023-12-27T22:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-27T22:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:04:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-27T22:04:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:04:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_26': 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13aca999df0a 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13aca999df0a 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:55 2023 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:55 2023 2023-12-27T22:04:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:55 2023 2023-12-27T22:04:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:04:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:04:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 48db3f6c-a1c5-4ad1-b8b2-0b426f65fc7a 2023-12-27T22:04:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:04:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:04:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-27T22:04:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:04:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:04:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:04:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:04:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T22:04:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:04:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:04:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 775168 bytes 2023-12-27T22:04:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 24583168 bytes 2023-12-27T22:04:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 60771328 bytes 2023-12-27T22:04:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 106831872 bytes 2023-12-27T22:04:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 154956800 bytes 2023-12-27T22:04:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 219509760 bytes 2023-12-27T22:04:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 282570752 bytes 2023-12-27T22:04:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 335312896 bytes 2023-12-27T22:04:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 379790336 bytes 2023-12-27T22:04:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 405441536 bytes 2023-12-27T22:04:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 427105280 bytes 2023-12-27T22:04:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 501980160 bytes 2023-12-27T22:04:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 565218304 bytes 2023-12-27T22:04:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 590247936 bytes 2023-12-27T22:04:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 622769152 bytes 2023-12-27T22:04:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 630422528 bytes 2023-12-27T22:04:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 679463936 bytes 2023-12-27T22:04:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 691528704 bytes 2023-12-27T22:04:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 740478976 bytes 2023-12-27T22:04:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 748685312 bytes 2023-12-27T22:04:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 825749504 bytes 2023-12-27T22:05:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 858050560 bytes 2023-12-27T22:05:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 902189056 bytes 2023-12-27T22:05:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 926305280 bytes 2023-12-27T22:05:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 974242816 bytes 2023-12-27T22:05:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 993361920 bytes 2023-12-27T22:05:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1055569920 bytes 2023-12-27T22:05:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1078631424 bytes 2023-12-27T22:05:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1118729216 bytes 2023-12-27T22:05:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1157713920 bytes 2023-12-27T22:05:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1249607680 bytes 2023-12-27T22:05:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1251602432 bytes 2023-12-27T22:05:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1293782016 bytes 2023-12-27T22:05:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1312165888 bytes 2023-12-27T22:05:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1335639040 bytes 2023-12-27T22:05:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1363346432 bytes 2023-12-27T22:05:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1411631104 bytes 2023-12-27T22:05:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1427020800 bytes 2023-12-27T22:05:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1455714304 bytes 2023-12-27T22:05:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1507312640 bytes 2023-12-27T22:05:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1557093376 bytes 2023-12-27T22:05:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1580118016 bytes 2023-12-27T22:05:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1616276480 bytes 2023-12-27T22:05:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1624687616 bytes 2023-12-27T22:05:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1684345856 bytes 2023-12-27T22:05:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1731523584 bytes 2023-12-27T22:05:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1745337344 bytes 2023-12-27T22:05:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1790844928 bytes 2023-12-27T22:05:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1828691968 bytes 2023-12-27T22:05:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1848023040 bytes 2023-12-27T22:05:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 1894672384 bytes 2023-12-27T22:05:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 1936863232 bytes 2023-12-27T22:05:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 1949776896 bytes 2023-12-27T22:05:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2027882496 bytes 2023-12-27T22:05:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2028511232 bytes 2023-12-27T22:05:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2091362304 bytes 2023-12-27T22:05:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2134962176 bytes 2023-12-27T22:05:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2185574400 bytes 2023-12-27T22:05:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2275838976 bytes 2023-12-27T22:05:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2315522048 bytes 2023-12-27T22:05:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2337720320 bytes 2023-12-27T22:05:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2373303296 bytes 2023-12-27T22:05:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2436286464 bytes 2023-12-27T22:05:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2460822528 bytes 2023-12-27T22:05:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2553629696 bytes 2023-12-27T22:05:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2576105472 bytes 2023-12-27T22:05:08.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2669223936 bytes 2023-12-27T22:05:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2787920896 bytes 2023-12-27T22:05:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2816902144 bytes 2023-12-27T22:05:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 2844370944 bytes 2023-12-27T22:05:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 2914663424 bytes 2023-12-27T22:05:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 2953997312 bytes 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_27 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:05:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:05:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:05:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2023-12-27T22:05:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:05:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-27T22:05:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:05:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-27T22:05:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:05:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-27T22:05:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_27': 2023-12-27T22:05:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:05:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:05:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13b8af78b927 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13b8af78b927 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:10 2023 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:10 2023 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:10 2023 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 98818c22-b08f-4edd-86c0-d236cb0457ef 2023-12-27T22:05:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:05:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:05:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-27T22:05:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:05:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:05:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:05:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:05:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:05:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 26885120 bytes 2023-12-27T22:05:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 38851584 bytes 2023-12-27T22:05:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 83539968 bytes 2023-12-27T22:05:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 102694912 bytes 2023-12-27T22:05:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 162363392 bytes 2023-12-27T22:05:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 214068224 bytes 2023-12-27T22:05:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 316218368 bytes 2023-12-27T22:05:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 317212672 bytes 2023-12-27T22:05:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 350966784 bytes 2023-12-27T22:05:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 363060224 bytes 2023-12-27T22:05:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 383006720 bytes 2023-12-27T22:05:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 417062912 bytes 2023-12-27T22:05:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 481611776 bytes 2023-12-27T22:05:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 543124480 bytes 2023-12-27T22:05:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 600736768 bytes 2023-12-27T22:05:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 601351168 bytes 2023-12-27T22:05:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 647272448 bytes 2023-12-27T22:05:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 699444224 bytes 2023-12-27T22:05:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 736208896 bytes 2023-12-27T22:05:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 812962816 bytes 2023-12-27T22:05:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 836411392 bytes 2023-12-27T22:05:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 872525824 bytes 2023-12-27T22:05:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 913244160 bytes 2023-12-27T22:05:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 940909568 bytes 2023-12-27T22:05:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 986573824 bytes 2023-12-27T22:05:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 988603392 bytes 2023-12-27T22:05:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1067693056 bytes 2023-12-27T22:05:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1086392320 bytes 2023-12-27T22:05:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1102734336 bytes 2023-12-27T22:05:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1127472128 bytes 2023-12-27T22:05:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1134636032 bytes 2023-12-27T22:05:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1136073728 bytes 2023-12-27T22:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1158052864 bytes 2023-12-27T22:05:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1171678208 bytes 2023-12-27T22:05:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1248515072 bytes 2023-12-27T22:05:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1309450240 bytes 2023-12-27T22:05:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1349598208 bytes 2023-12-27T22:05:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1437690880 bytes 2023-12-27T22:05:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1488716800 bytes 2023-12-27T22:05:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1545895936 bytes 2023-12-27T22:05:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1602277376 bytes 2023-12-27T22:05:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1602877440 bytes 2023-12-27T22:05:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1620773888 bytes 2023-12-27T22:05:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1693485056 bytes 2023-12-27T22:05:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1732416512 bytes 2023-12-27T22:05:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1748220928 bytes 2023-12-27T22:05:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1774246912 bytes 2023-12-27T22:05:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_28 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2023-12-27T22:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-27T22:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-27T22:05:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:05:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-27T22:05:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_28': 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13c4520ec8e8 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13c4520ec8e8 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:20 2023 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:20 2023 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:20 2023 2023-12-27T22:05:21.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:05:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:05:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 124b8604-3ea3-41f5-8bc5-de0e5cb86d1f 2023-12-27T22:05:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:05:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:05:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-27T22:05:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:05:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:05:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=11 2023-12-27T22:05:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:05:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:05:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 59186176 bytes 2023-12-27T22:05:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 139490304 bytes 2023-12-27T22:05:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 180508672 bytes 2023-12-27T22:05:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 194500608 bytes 2023-12-27T22:05:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 215787520 bytes 2023-12-27T22:05:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 300162048 bytes 2023-12-27T22:05:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 320834560 bytes 2023-12-27T22:05:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 362438656 bytes 2023-12-27T22:05:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 408582144 bytes 2023-12-27T22:05:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 432235520 bytes 2023-12-27T22:05:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 465162240 bytes 2023-12-27T22:05:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 485217280 bytes 2023-12-27T22:05:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 499240960 bytes 2023-12-27T22:05:23.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 517443584 bytes 2023-12-27T22:05:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 546745344 bytes 2023-12-27T22:05:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 579181568 bytes 2023-12-27T22:05:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 646221824 bytes 2023-12-27T22:05:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 677415936 bytes 2023-12-27T22:05:24.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 724833280 bytes 2023-12-27T22:05:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 742000640 bytes 2023-12-27T22:05:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 787037184 bytes 2023-12-27T22:05:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 817108992 bytes 2023-12-27T22:05:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 939057152 bytes 2023-12-27T22:05:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 940523520 bytes 2023-12-27T22:05:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 987543552 bytes 2023-12-27T22:05:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1000330240 bytes 2023-12-27T22:05:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1020646400 bytes 2023-12-27T22:05:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1080089600 bytes 2023-12-27T22:05:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1113493504 bytes 2023-12-27T22:05:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1124302848 bytes 2023-12-27T22:05:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1175358464 bytes 2023-12-27T22:05:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1250242560 bytes 2023-12-27T22:05:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1290472448 bytes 2023-12-27T22:05:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1291086848 bytes 2023-12-27T22:05:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1354604544 bytes 2023-12-27T22:05:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1368846336 bytes 2023-12-27T22:05:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1400328192 bytes 2023-12-27T22:05:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1445516288 bytes 2023-12-27T22:05:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1509839872 bytes 2023-12-27T22:05:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1536718848 bytes 2023-12-27T22:05:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1544232960 bytes 2023-12-27T22:05:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1605404672 bytes 2023-12-27T22:05:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1655948288 bytes 2023-12-27T22:05:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1669508096 bytes 2023-12-27T22:05:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1739081728 bytes 2023-12-27T22:05:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1843636224 bytes 2023-12-27T22:05:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1883417600 bytes 2023-12-27T22:05:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1908334592 bytes 2023-12-27T22:05:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1995795456 bytes 2023-12-27T22:05:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2026276864 bytes 2023-12-27T22:05:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2055661568 bytes 2023-12-27T22:05:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2113306624 bytes 2023-12-27T22:05:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2188088320 bytes 2023-12-27T22:05:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2200164352 bytes 2023-12-27T22:05:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2273869824 bytes 2023-12-27T22:05:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_29 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2023-12-27T22:05:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:05:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-27T22:05:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:05:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-27T22:05:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:05:33.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_29': 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13d0b710cfec 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13d0b710cfec 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:05:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:32 2023 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:32 2023 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:32 2023 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 916d4730-d7f2-40c8-a499-9224c313b98e 2023-12-27T22:05:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:05:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:05:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-27T22:05:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:05:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:05:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=13 2023-12-27T22:05:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:05:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:05:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 28898304 bytes 2023-12-27T22:05:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 97647616 bytes 2023-12-27T22:05:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 137690112 bytes 2023-12-27T22:05:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 213186560 bytes 2023-12-27T22:05:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 283873280 bytes 2023-12-27T22:05:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 346383360 bytes 2023-12-27T22:05:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 402739200 bytes 2023-12-27T22:05:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 430707712 bytes 2023-12-27T22:05:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 445528064 bytes 2023-12-27T22:05:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 558170112 bytes 2023-12-27T22:05:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 587860992 bytes 2023-12-27T22:05:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 610507776 bytes 2023-12-27T22:05:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 624086016 bytes 2023-12-27T22:05:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 660221952 bytes 2023-12-27T22:05:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 710247424 bytes 2023-12-27T22:05:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 713323520 bytes 2023-12-27T22:05:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 797438976 bytes 2023-12-27T22:05:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 835340288 bytes 2023-12-27T22:05:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 835973120 bytes 2023-12-27T22:05:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 836584448 bytes 2023-12-27T22:05:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 878071808 bytes 2023-12-27T22:05:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1008102400 bytes 2023-12-27T22:05:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1066056704 bytes 2023-12-27T22:05:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1094828032 bytes 2023-12-27T22:05:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1159148544 bytes 2023-12-27T22:05:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1218454528 bytes 2023-12-27T22:05:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1299046400 bytes 2023-12-27T22:05:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1329574912 bytes 2023-12-27T22:05:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1363899392 bytes 2023-12-27T22:05:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1399516160 bytes 2023-12-27T22:05:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1482467328 bytes 2023-12-27T22:05:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1483072512 bytes 2023-12-27T22:05:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1507265536 bytes 2023-12-27T22:05:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1566469120 bytes 2023-12-27T22:05:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1637637120 bytes 2023-12-27T22:05:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1707230208 bytes 2023-12-27T22:05:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1728381952 bytes 2023-12-27T22:05:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1783072768 bytes 2023-12-27T22:05:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1848429568 bytes 2023-12-27T22:05:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1905057792 bytes 2023-12-27T22:05:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1960872960 bytes 2023-12-27T22:05:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 2017992704 bytes 2023-12-27T22:05:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 2059913216 bytes 2023-12-27T22:05:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2074210304 bytes 2023-12-27T22:05:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2092081152 bytes 2023-12-27T22:05:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2179963904 bytes 2023-12-27T22:05:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2245066752 bytes 2023-12-27T22:05:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2274874368 bytes 2023-12-27T22:05:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2385658880 bytes 2023-12-27T22:05:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2430348288 bytes 2023-12-27T22:05:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2464351232 bytes 2023-12-27T22:05:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2464978944 bytes 2023-12-27T22:05:44.090 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T22:05:44.090+0000 7f78392c4700 -1 reset not still connected to 0x55c84217f1e0 2023-12-27T22:05:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2542911488 bytes 2023-12-27T22:05:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2562368512 bytes 2023-12-27T22:05:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2592367616 bytes 2023-12-27T22:05:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2704897024 bytes 2023-12-27T22:05:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2755830784 bytes 2023-12-27T22:05:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2816174080 bytes 2023-12-27T22:05:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2851356672 bytes 2023-12-27T22:05:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2968727552 bytes 2023-12-27T22:05:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 3003768832 bytes 2023-12-27T22:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:05:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_30 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2023-12-27T22:05:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:05:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-27T22:05:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:05:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-27T22:05:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:05:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-27T22:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_30': 2023-12-27T22:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13dca797c304 2023-12-27T22:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13dca797c304 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:46 2023 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:46 2023 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:46 2023 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 505598aa-3a40-4448-9218-b902cec83230 2023-12-27T22:05:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:05:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:05:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-27T22:05:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:05:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:05:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:05:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:05:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:05:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 47384576 bytes 2023-12-27T22:05:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 77712384 bytes 2023-12-27T22:05:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 197314560 bytes 2023-12-27T22:05:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 200407040 bytes 2023-12-27T22:05:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 244866048 bytes 2023-12-27T22:05:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 323072000 bytes 2023-12-27T22:05:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 396956672 bytes 2023-12-27T22:05:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 426614784 bytes 2023-12-27T22:05:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 508456960 bytes 2023-12-27T22:05:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 524581888 bytes 2023-12-27T22:05:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 552121344 bytes 2023-12-27T22:05:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 593844224 bytes 2023-12-27T22:05:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 710129664 bytes 2023-12-27T22:05:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 791235584 bytes 2023-12-27T22:05:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 831083520 bytes 2023-12-27T22:05:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 894175232 bytes 2023-12-27T22:05:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 926584832 bytes 2023-12-27T22:05:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 990406656 bytes 2023-12-27T22:05:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 1038790656 bytes 2023-12-27T22:05:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1050019840 bytes 2023-12-27T22:05:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1066012672 bytes 2023-12-27T22:05:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1112453120 bytes 2023-12-27T22:05:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1169788928 bytes 2023-12-27T22:05:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1220609024 bytes 2023-12-27T22:05:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1276746752 bytes 2023-12-27T22:05:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1320326144 bytes 2023-12-27T22:05:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1397872640 bytes 2023-12-27T22:05:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1455142912 bytes 2023-12-27T22:05:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1508213760 bytes 2023-12-27T22:05:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1564557312 bytes 2023-12-27T22:05:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1604783104 bytes 2023-12-27T22:05:54.167 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5479859277085781) > weight (0.25) 2023-12-27T22:05:54.167 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-27T22:05:54.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-27T22:05:54.168 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-27T22:05:54.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:05:54.170+0000 7f5f86428700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 54245) UID: 1000 2023-12-27T22:05:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1652916224 bytes 2023-12-27T22:05:54.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:05:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1673256960 bytes 2023-12-27T22:05:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1706739712 bytes 2023-12-27T22:05:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1707342848 bytes 2023-12-27T22:05:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1773576192 bytes 2023-12-27T22:05:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1840445440 bytes 2023-12-27T22:05:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1904660480 bytes 2023-12-27T22:05:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1982922752 bytes 2023-12-27T22:05:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1983678464 bytes 2023-12-27T22:05:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 2045050880 bytes 2023-12-27T22:05:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 2076206080 bytes 2023-12-27T22:05:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:05:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:05:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:05:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:05:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_31 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-27T22:05:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:05:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2023-12-27T22:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-27T22:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:05:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-27T22:05:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:05:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_31': 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 13e84a050159 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.13e84a050159 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:05:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:56 2023 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:56 2023 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:56 2023 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 103074c2-6a78-4270-a144-22bb49c87496 2023-12-27T22:05:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:05:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:05:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-27T22:05:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:05:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:05:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:05:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:05:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:05:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:05:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:05:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 43796480 bytes 2023-12-27T22:05:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 44422144 bytes 2023-12-27T22:05:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 85585920 bytes 2023-12-27T22:05:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 116515840 bytes 2023-12-27T22:05:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 144130048 bytes 2023-12-27T22:05:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 211768320 bytes 2023-12-27T22:05:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 284700672 bytes 2023-12-27T22:05:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 322429952 bytes 2023-12-27T22:05:59.079 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.896301901562691) > weight (0.25) 2023-12-27T22:05:59.080 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9766247085621037) > weight (0.25) 2023-12-27T22:05:59.080 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-27T22:05:59.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-27T22:05:59.080 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-27T22:05:59.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:2023-12-27T22:05:59.080+0000 7f5d72b28700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 46465) UID: 1000 2023-12-27T22:05:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 357947392 bytes 2023-12-27T22:05:59.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:05:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 418387968 bytes 2023-12-27T22:05:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 453016576 bytes 2023-12-27T22:05:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 516759552 bytes 2023-12-27T22:05:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 533678080 bytes 2023-12-27T22:06:00.169 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-27T22:06:00.169 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:06:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:06:00.170 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-27T22:06:00.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-27T22:06:00.170 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-27T22:06:00.172 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-27T22:06:00.173 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-27T22:06:00.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 634007552 bytes 2023-12-27T22:06:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 680511488 bytes 2023-12-27T22:06:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 737662976 bytes 2023-12-27T22:06:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 744022016 bytes 2023-12-27T22:06:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 803633152 bytes 2023-12-27T22:06:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 861452288 bytes 2023-12-27T22:06:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 870891520 bytes 2023-12-27T22:06:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 887670784 bytes 2023-12-27T22:06:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 926738432 bytes 2023-12-27T22:06:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1001696256 bytes 2023-12-27T22:06:02.081 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-27T22:06:02.081 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:06:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:06:02.082 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-27T22:06:02.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-27T22:06:02.082 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-27T22:06:02.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-27T22:06:02.085 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-12-27T22:06:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1020338176 bytes 2023-12-27T22:06:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1065708544 bytes 2023-12-27T22:06:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1087715328 bytes 2023-12-27T22:06:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1101085696 bytes 2023-12-27T22:06:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1107235840 bytes 2023-12-27T22:06:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1130027008 bytes 2023-12-27T22:06:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1130630144 bytes 2023-12-27T22:06:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1215810560 bytes 2023-12-27T22:06:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1251944448 bytes 2023-12-27T22:06:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1316947968 bytes 2023-12-27T22:06:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1349362688 bytes 2023-12-27T22:06:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1382912000 bytes 2023-12-27T22:06:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1442977792 bytes 2023-12-27T22:06:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1459602432 bytes 2023-12-27T22:06:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1519528960 bytes 2023-12-27T22:06:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1604704256 bytes 2023-12-27T22:06:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1644972032 bytes 2023-12-27T22:06:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1677616128 bytes 2023-12-27T22:06:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1700910080 bytes 2023-12-27T22:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1767901184 bytes 2023-12-27T22:06:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1802630144 bytes 2023-12-27T22:06:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1837266944 bytes 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_32 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-27T22:06:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:06:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:06:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:06:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:06:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2023-12-27T22:06:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:06:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-27T22:06:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:06:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-27T22:06:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:06:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_32': 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14031f42a2e 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14031f42a2e 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:06:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:06 2023 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:06 2023 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:06 2023 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 02d41175-0670-4c81-82ed-381c20b18bb9 2023-12-27T22:06:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:06:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:06:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-27T22:06:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:06:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:06:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=10 2023-12-27T22:06:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:06:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:06:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 16058368 bytes 2023-12-27T22:06:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 33305600 bytes 2023-12-27T22:06:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 55684096 bytes 2023-12-27T22:06:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 124914688 bytes 2023-12-27T22:06:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 168213504 bytes 2023-12-27T22:06:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 168838144 bytes 2023-12-27T22:06:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 188894208 bytes 2023-12-27T22:06:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 217980928 bytes 2023-12-27T22:06:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 259693568 bytes 2023-12-27T22:06:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 328672256 bytes 2023-12-27T22:06:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 400621568 bytes 2023-12-27T22:06:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 406558720 bytes 2023-12-27T22:06:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 459157504 bytes 2023-12-27T22:06:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 462582784 bytes 2023-12-27T22:06:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 525942784 bytes 2023-12-27T22:06:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 540452864 bytes 2023-12-27T22:06:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 592410624 bytes 2023-12-27T22:06:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 656882688 bytes 2023-12-27T22:06:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 713656320 bytes 2023-12-27T22:06:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 755824640 bytes 2023-12-27T22:06:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 817563648 bytes 2023-12-27T22:06:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 877977600 bytes 2023-12-27T22:06:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 972156928 bytes 2023-12-27T22:06:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 999461888 bytes 2023-12-27T22:06:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1025588224 bytes 2023-12-27T22:06:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1090254848 bytes 2023-12-27T22:06:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1204742144 bytes 2023-12-27T22:06:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1243623424 bytes 2023-12-27T22:06:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1272870912 bytes 2023-12-27T22:06:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1323551744 bytes 2023-12-27T22:06:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1332050944 bytes 2023-12-27T22:06:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1376962560 bytes 2023-12-27T22:06:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1389463552 bytes 2023-12-27T22:06:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1441357824 bytes 2023-12-27T22:06:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1457543168 bytes 2023-12-27T22:06:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1531509760 bytes 2023-12-27T22:06:13.843 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T22:06:13.843+0000 7f4f32c2f700 -1 reset not still connected to 0x559c1e0b9790 2023-12-27T22:06:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1564383232 bytes 2023-12-27T22:06:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1664482304 bytes 2023-12-27T22:06:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1722993664 bytes 2023-12-27T22:06:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1768835072 bytes 2023-12-27T22:06:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1880042496 bytes 2023-12-27T22:06:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1899985920 bytes 2023-12-27T22:06:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1923896320 bytes 2023-12-27T22:06:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1949712384 bytes 2023-12-27T22:06:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1986847744 bytes 2023-12-27T22:06:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2032063488 bytes 2023-12-27T22:06:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2136839168 bytes 2023-12-27T22:06:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2172706816 bytes 2023-12-27T22:06:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2267183104 bytes 2023-12-27T22:06:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:06:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:06:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:06:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:06:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:06:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_33 2023-12-27T22:06:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:06:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:06:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2023-12-27T22:06:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-27T22:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-27T22:06:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:06:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-27T22:06:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_33': 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 140fafdfa517 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.140fafdfa517 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:16 2023 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:16 2023 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:16 2023 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:06:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:06:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 232f1d52-1511-44c7-ac99-da55f617e6e4 2023-12-27T22:06:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:06:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:06:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-27T22:06:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:06:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:06:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T22:06:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:06:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:06:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 37027840 bytes 2023-12-27T22:06:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 75619328 bytes 2023-12-27T22:06:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 128016384 bytes 2023-12-27T22:06:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 176107520 bytes 2023-12-27T22:06:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 258217984 bytes 2023-12-27T22:06:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 303625216 bytes 2023-12-27T22:06:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 320679936 bytes 2023-12-27T22:06:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 381581312 bytes 2023-12-27T22:06:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 462523392 bytes 2023-12-27T22:06:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 521312256 bytes 2023-12-27T22:06:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 539623424 bytes 2023-12-27T22:06:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 570226688 bytes 2023-12-27T22:06:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 581604352 bytes 2023-12-27T22:06:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 657324032 bytes 2023-12-27T22:06:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 697011200 bytes 2023-12-27T22:06:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 775971840 bytes 2023-12-27T22:06:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 829358080 bytes 2023-12-27T22:06:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 877235200 bytes 2023-12-27T22:06:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 891694080 bytes 2023-12-27T22:06:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 940250112 bytes 2023-12-27T22:06:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 991183872 bytes 2023-12-27T22:06:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1034979328 bytes 2023-12-27T22:06:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1093924864 bytes 2023-12-27T22:06:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1104973824 bytes 2023-12-27T22:06:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1218581504 bytes 2023-12-27T22:06:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1221923840 bytes 2023-12-27T22:06:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1258110976 bytes 2023-12-27T22:06:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1317829632 bytes 2023-12-27T22:06:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1360649216 bytes 2023-12-27T22:06:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1399218176 bytes 2023-12-27T22:06:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1431605248 bytes 2023-12-27T22:06:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1452249088 bytes 2023-12-27T22:06:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1467627520 bytes 2023-12-27T22:06:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1478831104 bytes 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_34 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-27T22:06:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:06:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:06:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:06:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:06:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2023-12-27T22:06:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:06:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-27T22:06:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:06:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-27T22:06:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:06:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-27T22:06:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_34': 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 141b7bade53a 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.141b7bade53a 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:24 2023 2023-12-27T22:06:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:24 2023 2023-12-27T22:06:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:24 2023 2023-12-27T22:06:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:06:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:06:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 619c2405-7b4e-4332-9bd5-b1bd4115bc7d 2023-12-27T22:06:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:06:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:06:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-27T22:06:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:06:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:06:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T22:06:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:06:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:06:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 19182592 bytes 2023-12-27T22:06:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 69420032 bytes 2023-12-27T22:06:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 81693696 bytes 2023-12-27T22:06:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 138121216 bytes 2023-12-27T22:06:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 183927808 bytes 2023-12-27T22:06:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 191722496 bytes 2023-12-27T22:06:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 193103872 bytes 2023-12-27T22:06:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 271691776 bytes 2023-12-27T22:06:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 307721216 bytes 2023-12-27T22:06:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 308335616 bytes 2023-12-27T22:06:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 375482368 bytes 2023-12-27T22:06:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 449165312 bytes 2023-12-27T22:06:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 510298112 bytes 2023-12-27T22:06:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 540269568 bytes 2023-12-27T22:06:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 606567424 bytes 2023-12-27T22:06:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 668909568 bytes 2023-12-27T22:06:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 669532160 bytes 2023-12-27T22:06:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 686944256 bytes 2023-12-27T22:06:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 755809280 bytes 2023-12-27T22:06:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 782801920 bytes 2023-12-27T22:06:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 804983808 bytes 2023-12-27T22:06:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 866584576 bytes 2023-12-27T22:06:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 903201792 bytes 2023-12-27T22:06:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 969907200 bytes 2023-12-27T22:06:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1021635584 bytes 2023-12-27T22:06:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1043518464 bytes 2023-12-27T22:06:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1113622528 bytes 2023-12-27T22:06:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1157167104 bytes 2023-12-27T22:06:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1229265920 bytes 2023-12-27T22:06:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1250339840 bytes 2023-12-27T22:06:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1250960384 bytes 2023-12-27T22:06:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1385203712 bytes 2023-12-27T22:06:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1456083968 bytes 2023-12-27T22:06:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1456694272 bytes 2023-12-27T22:06:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1460861952 bytes 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_35 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:06:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:06:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2023-12-27T22:06:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:06:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-27T22:06:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:06:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-27T22:06:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:06:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-27T22:06:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_35': 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14272c3cb4bd 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14272c3cb4bd 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:32 2023 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:32 2023 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:32 2023 2023-12-27T22:06:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:06:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:06:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: abdbf0b7-80a9-450d-bd6c-9b78f7665b62 2023-12-27T22:06:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:06:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:06:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-27T22:06:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:06:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:06:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T22:06:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:06:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:06:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 35924992 bytes 2023-12-27T22:06:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 84402176 bytes 2023-12-27T22:06:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 101072896 bytes 2023-12-27T22:06:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 115645440 bytes 2023-12-27T22:06:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 163159040 bytes 2023-12-27T22:06:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 252829696 bytes 2023-12-27T22:06:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 320531456 bytes 2023-12-27T22:06:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 353295360 bytes 2023-12-27T22:06:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 412872704 bytes 2023-12-27T22:06:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 427461632 bytes 2023-12-27T22:06:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 451050496 bytes 2023-12-27T22:06:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 474377216 bytes 2023-12-27T22:06:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 519753728 bytes 2023-12-27T22:06:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 535800832 bytes 2023-12-27T22:06:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 555609088 bytes 2023-12-27T22:06:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 581304320 bytes 2023-12-27T22:06:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 603737088 bytes 2023-12-27T22:06:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 619611136 bytes 2023-12-27T22:06:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 620203008 bytes 2023-12-27T22:06:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 694144000 bytes 2023-12-27T22:06:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 717746176 bytes 2023-12-27T22:06:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 738689024 bytes 2023-12-27T22:06:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 776775680 bytes 2023-12-27T22:06:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 805690368 bytes 2023-12-27T22:06:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 876343296 bytes 2023-12-27T22:06:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 908691456 bytes 2023-12-27T22:06:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 911989760 bytes 2023-12-27T22:06:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 964246528 bytes 2023-12-27T22:06:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1034918912 bytes 2023-12-27T22:06:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1062232064 bytes 2023-12-27T22:06:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1139537920 bytes 2023-12-27T22:06:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1207071744 bytes 2023-12-27T22:06:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_36 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2023-12-27T22:06:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:06:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-27T22:06:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:06:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-27T22:06:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:06:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-27T22:06:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_36': 2023-12-27T22:06:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:06:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:06:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:06:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1433eb38d534 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1433eb38d534 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:39 2023 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:39 2023 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:39 2023 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d7e1cda4-b8f6-4e1c-9d85-6eaadd6f32c4 2023-12-27T22:06:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-27T22:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:06:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:06:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:06:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 24393728 bytes 2023-12-27T22:06:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 43523072 bytes 2023-12-27T22:06:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 64029696 bytes 2023-12-27T22:06:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 114511872 bytes 2023-12-27T22:06:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 132837376 bytes 2023-12-27T22:06:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 177464320 bytes 2023-12-27T22:06:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 230122496 bytes 2023-12-27T22:06:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 327767040 bytes 2023-12-27T22:06:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 378280960 bytes 2023-12-27T22:06:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 438498304 bytes 2023-12-27T22:06:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 478290944 bytes 2023-12-27T22:06:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 516644864 bytes 2023-12-27T22:06:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 607661056 bytes 2023-12-27T22:06:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 653150208 bytes 2023-12-27T22:06:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 653776896 bytes 2023-12-27T22:06:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 695231488 bytes 2023-12-27T22:06:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 734470144 bytes 2023-12-27T22:06:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 816799744 bytes 2023-12-27T22:06:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 939406336 bytes 2023-12-27T22:06:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 940038144 bytes 2023-12-27T22:06:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 993629184 bytes 2023-12-27T22:06:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1001440256 bytes 2023-12-27T22:06:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1008557056 bytes 2023-12-27T22:06:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1038310400 bytes 2023-12-27T22:06:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1041709056 bytes 2023-12-27T22:06:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1099819008 bytes 2023-12-27T22:06:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1124032512 bytes 2023-12-27T22:06:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1166193664 bytes 2023-12-27T22:06:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1182987264 bytes 2023-12-27T22:06:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1217319936 bytes 2023-12-27T22:06:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1244678144 bytes 2023-12-27T22:06:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1264993280 bytes 2023-12-27T22:06:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1310029824 bytes 2023-12-27T22:06:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1407396864 bytes 2023-12-27T22:06:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1460430848 bytes 2023-12-27T22:06:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1479983104 bytes 2023-12-27T22:06:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1480602624 bytes 2023-12-27T22:06:48.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1540613120 bytes 2023-12-27T22:06:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1567725568 bytes 2023-12-27T22:06:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1595540480 bytes 2023-12-27T22:06:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1635994624 bytes 2023-12-27T22:06:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1695609856 bytes 2023-12-27T22:06:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1759291392 bytes 2023-12-27T22:06:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1777716224 bytes 2023-12-27T22:06:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1809630208 bytes 2023-12-27T22:06:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1834510336 bytes 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_37 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-27T22:06:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:06:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:06:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:06:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:06:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2023-12-27T22:06:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:06:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-27T22:06:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:06:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-27T22:06:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:06:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-27T22:06:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_37': 2023-12-27T22:06:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:06:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:06:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:06:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 143f7500ffef 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.143f7500ffef 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:49 2023 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:49 2023 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:49 2023 2023-12-27T22:06:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:06:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:06:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: cb7251f0-99f4-4c03-a067-b3a87657c534 2023-12-27T22:06:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:06:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:06:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-27T22:06:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:06:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:06:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=5 2023-12-27T22:06:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:06:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:06:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 16091136 bytes 2023-12-27T22:06:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 52358144 bytes 2023-12-27T22:06:51.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 110134272 bytes 2023-12-27T22:06:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 141802496 bytes 2023-12-27T22:06:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 155369472 bytes 2023-12-27T22:06:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 193479680 bytes 2023-12-27T22:06:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 244583424 bytes 2023-12-27T22:06:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 297861120 bytes 2023-12-27T22:06:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 346979328 bytes 2023-12-27T22:06:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 412921856 bytes 2023-12-27T22:06:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 488346624 bytes 2023-12-27T22:06:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 514046976 bytes 2023-12-27T22:06:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 537217024 bytes 2023-12-27T22:06:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 596519936 bytes 2023-12-27T22:06:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 650192896 bytes 2023-12-27T22:06:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 728271872 bytes 2023-12-27T22:06:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 812178432 bytes 2023-12-27T22:06:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 843392000 bytes 2023-12-27T22:06:55.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 887419904 bytes 2023-12-27T22:06:55.096 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T22:06:55.095+0000 7f78392c4700 -1 reset not still connected to 0x55c851ae9520 2023-12-27T22:06:55.098 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T22:06:55.097+0000 7f78392c4700 -1 reset not still connected to 0x55c851ca9e10 2023-12-27T22:06:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 892078080 bytes 2023-12-27T22:06:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 945287168 bytes 2023-12-27T22:06:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 989068288 bytes 2023-12-27T22:06:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1031615488 bytes 2023-12-27T22:06:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1035597824 bytes 2023-12-27T22:06:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:06:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_38 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2023-12-27T22:06:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:06:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-27T22:06:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:06:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:06:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-27T22:06:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:06:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-27T22:06:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_38': 2023-12-27T22:06:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:06:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:06:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:06:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 144bb6364277 2023-12-27T22:06:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.144bb6364277 2023-12-27T22:06:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:56 2023 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:56 2023 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:56 2023 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: a570017e-0f25-4b1e-bf55-91e5287082df 2023-12-27T22:06:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:06:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:06:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-27T22:06:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:06:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:06:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:06:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:06:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=11 2023-12-27T22:06:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:06:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:06:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 63536128 bytes 2023-12-27T22:06:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 131802112 bytes 2023-12-27T22:06:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 147435520 bytes 2023-12-27T22:06:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 193885184 bytes 2023-12-27T22:06:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 208224256 bytes 2023-12-27T22:06:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 233803776 bytes 2023-12-27T22:06:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 254531584 bytes 2023-12-27T22:06:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 330567680 bytes 2023-12-27T22:06:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 366426112 bytes 2023-12-27T22:06:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 396135424 bytes 2023-12-27T22:06:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 433945600 bytes 2023-12-27T22:06:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 477655040 bytes 2023-12-27T22:06:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 516610048 bytes 2023-12-27T22:06:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 555740160 bytes 2023-12-27T22:06:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 603424768 bytes 2023-12-27T22:07:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 655407104 bytes 2023-12-27T22:07:00.296 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.293+0000 7fc355a18700 -1 reset not still connected to 0x55deea2e7d40 2023-12-27T22:07:00.296 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.293+0000 7fc355a18700 -1 reset not still connected to 0x55deef866a90 2023-12-27T22:07:00.301 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55defd6c4000 2023-12-27T22:07:00.301 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55defd6c5110 2023-12-27T22:07:00.301 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55defee9a1a0 2023-12-27T22:07:00.302 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55df03f8f5f0 2023-12-27T22:07:00.302 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55df07934680 2023-12-27T22:07:00.302 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55df07d821a0 2023-12-27T22:07:00.302 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55df07d829c0 2023-12-27T22:07:00.302 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55df09408000 2023-12-27T22:07:00.302 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:07:00.301+0000 7fc355a18700 -1 reset not still connected to 0x55df1721dad0 2023-12-27T22:07:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 740985856 bytes 2023-12-27T22:07:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 779208704 bytes 2023-12-27T22:07:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 797509632 bytes 2023-12-27T22:07:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 817017856 bytes 2023-12-27T22:07:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 886375424 bytes 2023-12-27T22:07:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 930070528 bytes 2023-12-27T22:07:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 966500352 bytes 2023-12-27T22:07:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1022441472 bytes 2023-12-27T22:07:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1061445632 bytes 2023-12-27T22:07:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1171887104 bytes 2023-12-27T22:07:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1248792576 bytes 2023-12-27T22:07:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1302404096 bytes 2023-12-27T22:07:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1344235520 bytes 2023-12-27T22:07:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1420216320 bytes 2023-12-27T22:07:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1437338624 bytes 2023-12-27T22:07:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1469282304 bytes 2023-12-27T22:07:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1502938112 bytes 2023-12-27T22:07:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1574580224 bytes 2023-12-27T22:07:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1610883072 bytes 2023-12-27T22:07:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1645706240 bytes 2023-12-27T22:07:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1711369216 bytes 2023-12-27T22:07:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1725604864 bytes 2023-12-27T22:07:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1757359104 bytes 2023-12-27T22:07:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1775221760 bytes 2023-12-27T22:07:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1820506112 bytes 2023-12-27T22:07:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1879035904 bytes 2023-12-27T22:07:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1915984896 bytes 2023-12-27T22:07:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1927615488 bytes 2023-12-27T22:07:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1966313472 bytes 2023-12-27T22:07:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2053111808 bytes 2023-12-27T22:07:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2080248832 bytes 2023-12-27T22:07:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2101035008 bytes 2023-12-27T22:07:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2135189504 bytes 2023-12-27T22:07:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2178994176 bytes 2023-12-27T22:07:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2196199424 bytes 2023-12-27T22:07:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2227970048 bytes 2023-12-27T22:07:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2265333760 bytes 2023-12-27T22:07:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2324762624 bytes 2023-12-27T22:07:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2382702592 bytes 2023-12-27T22:07:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:07:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:07:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:07:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:07:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:07:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_39 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2023-12-27T22:07:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:07:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-27T22:07:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:07:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-27T22:07:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:07:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-27T22:07:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_39': 2023-12-27T22:07:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:07:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:07:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14572dad4557 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14572dad4557 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:08 2023 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:08 2023 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:08 2023 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 4cb400ed-6f6e-497a-ba79-d1988e890173 2023-12-27T22:07:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:07:09.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:07:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-27T22:07:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:07:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:07:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:07:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:07:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:07:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 30695424 bytes 2023-12-27T22:07:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 54889472 bytes 2023-12-27T22:07:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 114813952 bytes 2023-12-27T22:07:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 156313600 bytes 2023-12-27T22:07:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 192757760 bytes 2023-12-27T22:07:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 242848768 bytes 2023-12-27T22:07:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 266771456 bytes 2023-12-27T22:07:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 289456128 bytes 2023-12-27T22:07:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 334125056 bytes 2023-12-27T22:07:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 383415296 bytes 2023-12-27T22:07:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 465266688 bytes 2023-12-27T22:07:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 507841536 bytes 2023-12-27T22:07:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 533285888 bytes 2023-12-27T22:07:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 610979840 bytes 2023-12-27T22:07:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 663242752 bytes 2023-12-27T22:07:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 778545152 bytes 2023-12-27T22:07:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 809214976 bytes 2023-12-27T22:07:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 841078784 bytes 2023-12-27T22:07:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 872415232 bytes 2023-12-27T22:07:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 881823744 bytes 2023-12-27T22:07:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 911347712 bytes 2023-12-27T22:07:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 957649920 bytes 2023-12-27T22:07:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1007034368 bytes 2023-12-27T22:07:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1064313856 bytes 2023-12-27T22:07:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1076209664 bytes 2023-12-27T22:07:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1097966592 bytes 2023-12-27T22:07:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1166262272 bytes 2023-12-27T22:07:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1187260416 bytes 2023-12-27T22:07:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1245682688 bytes 2023-12-27T22:07:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1280089088 bytes 2023-12-27T22:07:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1319388160 bytes 2023-12-27T22:07:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1320012800 bytes 2023-12-27T22:07:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1320643584 bytes 2023-12-27T22:07:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1352924160 bytes 2023-12-27T22:07:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1408817152 bytes 2023-12-27T22:07:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1441242112 bytes 2023-12-27T22:07:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1452052480 bytes 2023-12-27T22:07:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1545767936 bytes 2023-12-27T22:07:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1575825408 bytes 2023-12-27T22:07:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1613051904 bytes 2023-12-27T22:07:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1673442304 bytes 2023-12-27T22:07:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1709965312 bytes 2023-12-27T22:07:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1767591936 bytes 2023-12-27T22:07:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1812385792 bytes 2023-12-27T22:07:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1824831488 bytes 2023-12-27T22:07:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1898375168 bytes 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_40 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:07:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2023-12-27T22:07:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:07:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-27T22:07:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:07:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-27T22:07:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:07:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-27T22:07:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_40': 2023-12-27T22:07:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1463730ea265 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1463730ea265 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:18 2023 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:18 2023 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:18 2023 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 0b9a6172-e596-4e63-ada8-38dbbc7d188d 2023-12-27T22:07:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:07:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:07:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-27T22:07:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:07:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:07:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T22:07:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:07:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:07:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 20324352 bytes 2023-12-27T22:07:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 54886400 bytes 2023-12-27T22:07:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 84399104 bytes 2023-12-27T22:07:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 96866304 bytes 2023-12-27T22:07:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 103461888 bytes 2023-12-27T22:07:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 138852352 bytes 2023-12-27T22:07:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 153417728 bytes 2023-12-27T22:07:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 218288128 bytes 2023-12-27T22:07:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 305609728 bytes 2023-12-27T22:07:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 312681472 bytes 2023-12-27T22:07:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 401056768 bytes 2023-12-27T22:07:21.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 421638144 bytes 2023-12-27T22:07:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 485184512 bytes 2023-12-27T22:07:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 572161024 bytes 2023-12-27T22:07:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 641019904 bytes 2023-12-27T22:07:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 691663872 bytes 2023-12-27T22:07:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 739549184 bytes 2023-12-27T22:07:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 817038336 bytes 2023-12-27T22:07:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 834573312 bytes 2023-12-27T22:07:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 849236992 bytes 2023-12-27T22:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 869076992 bytes 2023-12-27T22:07:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 944677888 bytes 2023-12-27T22:07:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 960557056 bytes 2023-12-27T22:07:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 969053184 bytes 2023-12-27T22:07:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 969699328 bytes 2023-12-27T22:07:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 985435136 bytes 2023-12-27T22:07:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1088498688 bytes 2023-12-27T22:07:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1169844224 bytes 2023-12-27T22:07:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1188689920 bytes 2023-12-27T22:07:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1273622528 bytes 2023-12-27T22:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:07:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_41 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2023-12-27T22:07:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:07:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-27T22:07:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:07:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-27T22:07:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:07:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_41': 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 146f4537f7fa 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.146f4537f7fa 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:25 2023 2023-12-27T22:07:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:25 2023 2023-12-27T22:07:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:25 2023 2023-12-27T22:07:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:07:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:07:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 8f628a70-f2b5-4041-a2d4-bbbc883caac7 2023-12-27T22:07:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:07:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:07:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-27T22:07:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:07:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:07:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:07:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:07:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:07:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 613376 bytes 2023-12-27T22:07:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 105339904 bytes 2023-12-27T22:07:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 147798016 bytes 2023-12-27T22:07:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 191172608 bytes 2023-12-27T22:07:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 252196864 bytes 2023-12-27T22:07:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 310141952 bytes 2023-12-27T22:07:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 353983488 bytes 2023-12-27T22:07:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 426441728 bytes 2023-12-27T22:07:28.174 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.25565751621488586) > weight (0.25) 2023-12-27T22:07:28.174 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8146738426418506) > weight (0.25) 2023-12-27T22:07:28.174 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3948421497064607) > weight (0.25) 2023-12-27T22:07:28.174 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-27T22:07:28.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-27T22:07:28.174 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-27T22:07:28.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi196.stderr:2023-12-27T22:07:28.176+0000 7f7dd0af9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 46588) UID: 1000 2023-12-27T22:07:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 441019392 bytes 2023-12-27T22:07:28.383 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:07:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 493149184 bytes 2023-12-27T22:07:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 547261440 bytes 2023-12-27T22:07:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 606664704 bytes 2023-12-27T22:07:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 626667520 bytes 2023-12-27T22:07:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 733479936 bytes 2023-12-27T22:07:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 749135872 bytes 2023-12-27T22:07:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 749755392 bytes 2023-12-27T22:07:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 766597120 bytes 2023-12-27T22:07:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 851070976 bytes 2023-12-27T22:07:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 890274816 bytes 2023-12-27T22:07:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 962972672 bytes 2023-12-27T22:07:30.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 987879424 bytes 2023-12-27T22:07:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1021413376 bytes 2023-12-27T22:07:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1063593984 bytes 2023-12-27T22:07:31.175 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-27T22:07:31.175 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:07:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:07:31.176 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-27T22:07:31.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-27T22:07:31.176 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-27T22:07:31.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-27T22:07:31.180 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-27T22:07:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1127792640 bytes 2023-12-27T22:07:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1185863680 bytes 2023-12-27T22:07:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1230035968 bytes 2023-12-27T22:07:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1248155648 bytes 2023-12-27T22:07:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1261021184 bytes 2023-12-27T22:07:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1281937408 bytes 2023-12-27T22:07:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1371264000 bytes 2023-12-27T22:07:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1446738944 bytes 2023-12-27T22:07:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1499660288 bytes 2023-12-27T22:07:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1511786496 bytes 2023-12-27T22:07:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1591496704 bytes 2023-12-27T22:07:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1646168064 bytes 2023-12-27T22:07:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1706729472 bytes 2023-12-27T22:07:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1726641152 bytes 2023-12-27T22:07:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1738824704 bytes 2023-12-27T22:07:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1753111552 bytes 2023-12-27T22:07:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1794915328 bytes 2023-12-27T22:07:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1803279360 bytes 2023-12-27T22:07:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1804991488 bytes 2023-12-27T22:07:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1841193984 bytes 2023-12-27T22:07:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1899498496 bytes 2023-12-27T22:07:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1900091392 bytes 2023-12-27T22:07:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:07:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:07:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:07:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_42 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2023-12-27T22:07:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:07:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-27T22:07:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:07:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-27T22:07:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:07:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_42': 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14832cabc44d 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14832cabc44d 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:35 2023 2023-12-27T22:07:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:35 2023 2023-12-27T22:07:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:35 2023 2023-12-27T22:07:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:07:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:07:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 7b5df11a-e491-4959-be66-344df8a2e44a 2023-12-27T22:07:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:07:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:07:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-27T22:07:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:07:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:07:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=13 2023-12-27T22:07:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:07:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:07:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 49120256 bytes 2023-12-27T22:07:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 116188160 bytes 2023-12-27T22:07:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 119593984 bytes 2023-12-27T22:07:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 182576128 bytes 2023-12-27T22:07:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 201646080 bytes 2023-12-27T22:07:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 221903872 bytes 2023-12-27T22:07:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 240954368 bytes 2023-12-27T22:07:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 257735680 bytes 2023-12-27T22:07:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 297665536 bytes 2023-12-27T22:07:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 308729856 bytes 2023-12-27T22:07:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 353368064 bytes 2023-12-27T22:07:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 422695936 bytes 2023-12-27T22:07:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 473399296 bytes 2023-12-27T22:07:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 513094656 bytes 2023-12-27T22:07:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 537340928 bytes 2023-12-27T22:07:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 582354944 bytes 2023-12-27T22:07:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 631706624 bytes 2023-12-27T22:07:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 664685568 bytes 2023-12-27T22:07:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 786082816 bytes 2023-12-27T22:07:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 806097920 bytes 2023-12-27T22:07:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 831223808 bytes 2023-12-27T22:07:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 891790336 bytes 2023-12-27T22:07:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 903119872 bytes 2023-12-27T22:07:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 918050816 bytes 2023-12-27T22:07:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 947376128 bytes 2023-12-27T22:07:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 958043136 bytes 2023-12-27T22:07:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 981961728 bytes 2023-12-27T22:07:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1052169216 bytes 2023-12-27T22:07:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1091382272 bytes 2023-12-27T22:07:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1151675392 bytes 2023-12-27T22:07:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1211995136 bytes 2023-12-27T22:07:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1285455872 bytes 2023-12-27T22:07:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1321519104 bytes 2023-12-27T22:07:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1377303552 bytes 2023-12-27T22:07:43.085 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9321900927630212) > weight (0.25) 2023-12-27T22:07:43.085 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7224915799838243) > weight (0.25) 2023-12-27T22:07:43.085 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-27T22:07:43.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-27T22:07:43.086 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T22:07:43.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:2023-12-27T22:07:43.087+0000 7f9cb1e72700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 59997) UID: 1000 2023-12-27T22:07:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1416083456 bytes 2023-12-27T22:07:43.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:07:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1489871872 bytes 2023-12-27T22:07:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1513563136 bytes 2023-12-27T22:07:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1572513792 bytes 2023-12-27T22:07:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1595656192 bytes 2023-12-27T22:07:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1639513088 bytes 2023-12-27T22:07:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1652262912 bytes 2023-12-27T22:07:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1711998976 bytes 2023-12-27T22:07:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1753706496 bytes 2023-12-27T22:07:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1828547584 bytes 2023-12-27T22:07:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1878792192 bytes 2023-12-27T22:07:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1931569152 bytes 2023-12-27T22:07:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2061731840 bytes 2023-12-27T22:07:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2130055168 bytes 2023-12-27T22:07:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2152862720 bytes 2023-12-27T22:07:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2164966400 bytes 2023-12-27T22:07:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2177575936 bytes 2023-12-27T22:07:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2207385600 bytes 2023-12-27T22:07:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2265254912 bytes 2023-12-27T22:07:47.087 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-27T22:07:47.087 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:07:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:07:47.087 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-27T22:07:47.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-27T22:07:47.088 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-27T22:07:47.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-27T22:07:47.090 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-12-27T22:07:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2314138624 bytes 2023-12-27T22:07:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2314750976 bytes 2023-12-27T22:07:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2336022528 bytes 2023-12-27T22:07:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2387134464 bytes 2023-12-27T22:07:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2424721408 bytes 2023-12-27T22:07:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2442497024 bytes 2023-12-27T22:07:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2466828288 bytes 2023-12-27T22:07:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2507404288 bytes 2023-12-27T22:07:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2524988416 bytes 2023-12-27T22:07:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2565780480 bytes 2023-12-27T22:07:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2589924352 bytes 2023-12-27T22:07:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2591409152 bytes 2023-12-27T22:07:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2627929088 bytes 2023-12-27T22:07:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:07:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:07:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:07:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:07:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_43 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:07:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2023-12-27T22:07:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:07:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-27T22:07:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:07:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:07:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-27T22:07:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:07:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_43': 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1494f533f0aa 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1494f533f0aa 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:07:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:49 2023 2023-12-27T22:07:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:49 2023 2023-12-27T22:07:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:49 2023 2023-12-27T22:07:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:07:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:07:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 6303d43d-7367-464b-9985-add1a3b57264 2023-12-27T22:07:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:07:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:07:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-27T22:07:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:07:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:07:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:07:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:07:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=10 2023-12-27T22:07:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:07:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:07:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 19992576 bytes 2023-12-27T22:07:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 70751232 bytes 2023-12-27T22:07:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 91311104 bytes 2023-12-27T22:07:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 175697920 bytes 2023-12-27T22:07:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 186632192 bytes 2023-12-27T22:07:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 219803648 bytes 2023-12-27T22:07:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 273617920 bytes 2023-12-27T22:07:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 280312832 bytes 2023-12-27T22:07:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 280946688 bytes 2023-12-27T22:07:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 335990784 bytes 2023-12-27T22:07:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 382013440 bytes 2023-12-27T22:07:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 433586176 bytes 2023-12-27T22:07:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 483694592 bytes 2023-12-27T22:07:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 528305152 bytes 2023-12-27T22:07:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 566600704 bytes 2023-12-27T22:07:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 592974848 bytes 2023-12-27T22:07:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 667342848 bytes 2023-12-27T22:07:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 690956288 bytes 2023-12-27T22:07:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 804571136 bytes 2023-12-27T22:07:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 814839808 bytes 2023-12-27T22:07:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 887762944 bytes 2023-12-27T22:07:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 930418688 bytes 2023-12-27T22:07:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 972250112 bytes 2023-12-27T22:07:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 987231232 bytes 2023-12-27T22:07:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1009531904 bytes 2023-12-27T22:07:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1046545408 bytes 2023-12-27T22:07:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1057267712 bytes 2023-12-27T22:07:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1112971264 bytes 2023-12-27T22:07:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1139283968 bytes 2023-12-27T22:07:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1144441856 bytes 2023-12-27T22:07:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1207540736 bytes 2023-12-27T22:07:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1265428480 bytes 2023-12-27T22:07:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1275011072 bytes 2023-12-27T22:07:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1358255104 bytes 2023-12-27T22:07:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1454981120 bytes 2023-12-27T22:07:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1552517120 bytes 2023-12-27T22:07:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1595932672 bytes 2023-12-27T22:07:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1714422784 bytes 2023-12-27T22:07:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1754761216 bytes 2023-12-27T22:07:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1779137536 bytes 2023-12-27T22:07:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1830587392 bytes 2023-12-27T22:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1888416768 bytes 2023-12-27T22:07:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1933459456 bytes 2023-12-27T22:07:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1983455232 bytes 2023-12-27T22:07:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2011443200 bytes 2023-12-27T22:07:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2074682368 bytes 2023-12-27T22:07:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2103462912 bytes 2023-12-27T22:08:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2107031552 bytes 2023-12-27T22:08:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2122977280 bytes 2023-12-27T22:08:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:08:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:08:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:08:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:08:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_44 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-27T22:08:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2023-12-27T22:08:00.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:08:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-27T22:08:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:08:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-27T22:08:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:08:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-27T22:08:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_44': 2023-12-27T22:08:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:08:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:08:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:08:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14a0fa117e32 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14a0fa117e32 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:00 2023 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:00 2023 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:00 2023 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 5ce59573-6607-4c36-b8ca-ffaa5d72c073 2023-12-27T22:08:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:08:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:08:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-27T22:08:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:08:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:08:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:08:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:08:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:08:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 25466880 bytes 2023-12-27T22:08:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 60823552 bytes 2023-12-27T22:08:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 86350848 bytes 2023-12-27T22:08:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 86959104 bytes 2023-12-27T22:08:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 118907904 bytes 2023-12-27T22:08:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 130230272 bytes 2023-12-27T22:08:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 182239232 bytes 2023-12-27T22:08:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 213396480 bytes 2023-12-27T22:08:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 239599616 bytes 2023-12-27T22:08:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 295116800 bytes 2023-12-27T22:08:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 311693312 bytes 2023-12-27T22:08:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 364854272 bytes 2023-12-27T22:08:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 403678208 bytes 2023-12-27T22:08:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 426759168 bytes 2023-12-27T22:08:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 525049856 bytes 2023-12-27T22:08:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 545746944 bytes 2023-12-27T22:08:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 574271488 bytes 2023-12-27T22:08:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 654433280 bytes 2023-12-27T22:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 670814208 bytes 2023-12-27T22:08:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 764042240 bytes 2023-12-27T22:08:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 766350336 bytes 2023-12-27T22:08:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 817260544 bytes 2023-12-27T22:08:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 895956992 bytes 2023-12-27T22:08:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 996411392 bytes 2023-12-27T22:08:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1021376512 bytes 2023-12-27T22:08:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1103766528 bytes 2023-12-27T22:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1145472000 bytes 2023-12-27T22:08:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1170933760 bytes 2023-12-27T22:08:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1207224320 bytes 2023-12-27T22:08:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1208717312 bytes 2023-12-27T22:08:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1257543680 bytes 2023-12-27T22:08:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1277082624 bytes 2023-12-27T22:08:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1277703168 bytes 2023-12-27T22:08:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1294064640 bytes 2023-12-27T22:08:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1319690240 bytes 2023-12-27T22:08:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1368299520 bytes 2023-12-27T22:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1390777344 bytes 2023-12-27T22:08:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1433154560 bytes 2023-12-27T22:08:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1494861824 bytes 2023-12-27T22:08:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1583125504 bytes 2023-12-27T22:08:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1633282048 bytes 2023-12-27T22:08:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1665912832 bytes 2023-12-27T22:08:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1811664896 bytes 2023-12-27T22:08:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1836977152 bytes 2023-12-27T22:08:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1861282816 bytes 2023-12-27T22:08:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:08:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_45 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:08:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2023-12-27T22:08:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:08:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-27T22:08:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:08:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-27T22:08:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:08:10.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-27T22:08:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_45': 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14ac7fabfbfa 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14ac7fabfbfa 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:09 2023 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:09 2023 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:09 2023 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:08:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:08:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: c1569f06-8c36-41d6-83d4-36589c3c6c54 2023-12-27T22:08:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:08:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:08:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-27T22:08:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:08:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:08:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T22:08:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:08:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:08:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 103258112 bytes 2023-12-27T22:08:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 128312320 bytes 2023-12-27T22:08:11.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 185216000 bytes 2023-12-27T22:08:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 273581056 bytes 2023-12-27T22:08:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 305400832 bytes 2023-12-27T22:08:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 323380224 bytes 2023-12-27T22:08:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 342767616 bytes 2023-12-27T22:08:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 415278080 bytes 2023-12-27T22:08:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 441677824 bytes 2023-12-27T22:08:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 490985472 bytes 2023-12-27T22:08:13.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 522868736 bytes 2023-12-27T22:08:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 597774336 bytes 2023-12-27T22:08:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 658439168 bytes 2023-12-27T22:08:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 691000320 bytes 2023-12-27T22:08:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 691600384 bytes 2023-12-27T22:08:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 704897024 bytes 2023-12-27T22:08:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 738841600 bytes 2023-12-27T22:08:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 795500544 bytes 2023-12-27T22:08:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 856087552 bytes 2023-12-27T22:08:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 868406272 bytes 2023-12-27T22:08:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 940705792 bytes 2023-12-27T22:08:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1035895808 bytes 2023-12-27T22:08:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1119128576 bytes 2023-12-27T22:08:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1206794240 bytes 2023-12-27T22:08:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1228549120 bytes 2023-12-27T22:08:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1266717696 bytes 2023-12-27T22:08:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1275521024 bytes 2023-12-27T22:08:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1308888064 bytes 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_46 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-27T22:08:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:08:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:08:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:08:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:08:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2023-12-27T22:08:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:08:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-27T22:08:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:08:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-27T22:08:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:08:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-27T22:08:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_46': 2023-12-27T22:08:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:08:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:08:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14b8127ba107 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14b8127ba107 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:16 2023 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:16 2023 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:16 2023 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 79fc8fc4-a92b-41e3-8510-d70489a1905a 2023-12-27T22:08:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:08:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:08:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-27T22:08:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:08:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:08:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T22:08:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:08:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:08:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 19414016 bytes 2023-12-27T22:08:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 27992064 bytes 2023-12-27T22:08:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 103341056 bytes 2023-12-27T22:08:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 136641536 bytes 2023-12-27T22:08:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 150068224 bytes 2023-12-27T22:08:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 150700032 bytes 2023-12-27T22:08:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 192721920 bytes 2023-12-27T22:08:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 236536832 bytes 2023-12-27T22:08:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 281164800 bytes 2023-12-27T22:08:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 348838912 bytes 2023-12-27T22:08:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 389133312 bytes 2023-12-27T22:08:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 441155584 bytes 2023-12-27T22:08:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 459192320 bytes 2023-12-27T22:08:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 492882944 bytes 2023-12-27T22:08:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 543174656 bytes 2023-12-27T22:08:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 606330880 bytes 2023-12-27T22:08:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 606953472 bytes 2023-12-27T22:08:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 662968320 bytes 2023-12-27T22:08:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 733933568 bytes 2023-12-27T22:08:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 792686592 bytes 2023-12-27T22:08:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 816158720 bytes 2023-12-27T22:08:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 826660864 bytes 2023-12-27T22:08:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 912894976 bytes 2023-12-27T22:08:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 958412800 bytes 2023-12-27T22:08:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 978930688 bytes 2023-12-27T22:08:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1087208448 bytes 2023-12-27T22:08:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1166818304 bytes 2023-12-27T22:08:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1197347840 bytes 2023-12-27T22:08:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1233410048 bytes 2023-12-27T22:08:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1268563968 bytes 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_47 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:08:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2023-12-27T22:08:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:08:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-27T22:08:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:08:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-27T22:08:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:08:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_47': 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14c4e9379b59 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14c4e9379b59 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:08:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:23 2023 2023-12-27T22:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:23 2023 2023-12-27T22:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:23 2023 2023-12-27T22:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: b49aa2f8-63aa-40d1-91ed-40c882361a94 2023-12-27T22:08:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:08:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:08:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-27T22:08:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:08:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:08:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=8 2023-12-27T22:08:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:08:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:08:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 618496 bytes 2023-12-27T22:08:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 17301504 bytes 2023-12-27T22:08:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 69686272 bytes 2023-12-27T22:08:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 95047680 bytes 2023-12-27T22:08:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 150950912 bytes 2023-12-27T22:08:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 173855744 bytes 2023-12-27T22:08:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 198618112 bytes 2023-12-27T22:08:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 257625088 bytes 2023-12-27T22:08:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 292269056 bytes 2023-12-27T22:08:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 364164096 bytes 2023-12-27T22:08:26.879 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:08:26.879+0000 7fc355a18700 -1 reset not still connected to 0x55deefba8d00 2023-12-27T22:08:26.879 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:08:26.879+0000 7fc355a18700 -1 reset not still connected to 0x55defd6c4000 2023-12-27T22:08:26.879 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:08:26.879+0000 7fc355a18700 -1 reset not still connected to 0x55defd6c5110 2023-12-27T22:08:26.879 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:08:26.879+0000 7fc355a18700 -1 reset not still connected to 0x55df03f8f5f0 2023-12-27T22:08:26.880 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:08:26.879+0000 7fc355a18700 -1 reset not still connected to 0x55df07934680 2023-12-27T22:08:26.882 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:08:26.879+0000 7fc355a18700 -1 reset not still connected to 0x55df07d6f110 2023-12-27T22:08:26.882 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:08:26.879+0000 7fc355a18700 -1 reset not still connected to 0x55df07d829c0 2023-12-27T22:08:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 367682560 bytes 2023-12-27T22:08:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 429268992 bytes 2023-12-27T22:08:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 515008512 bytes 2023-12-27T22:08:27.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 541956096 bytes 2023-12-27T22:08:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 589919232 bytes 2023-12-27T22:08:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 615802880 bytes 2023-12-27T22:08:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 668002304 bytes 2023-12-27T22:08:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 715069440 bytes 2023-12-27T22:08:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 762404864 bytes 2023-12-27T22:08:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 812013568 bytes 2023-12-27T22:08:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 812645376 bytes 2023-12-27T22:08:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 835402752 bytes 2023-12-27T22:08:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 892675072 bytes 2023-12-27T22:08:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 893298688 bytes 2023-12-27T22:08:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 933432320 bytes 2023-12-27T22:08:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1037611008 bytes 2023-12-27T22:08:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1066947584 bytes 2023-12-27T22:08:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1113550848 bytes 2023-12-27T22:08:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1151303680 bytes 2023-12-27T22:08:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1191674880 bytes 2023-12-27T22:08:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1222022144 bytes 2023-12-27T22:08:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1242995712 bytes 2023-12-27T22:08:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1274625024 bytes 2023-12-27T22:08:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1325083648 bytes 2023-12-27T22:08:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1371043840 bytes 2023-12-27T22:08:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1433264128 bytes 2023-12-27T22:08:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1493478400 bytes 2023-12-27T22:08:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1570281472 bytes 2023-12-27T22:08:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1593780224 bytes 2023-12-27T22:08:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1618055168 bytes 2023-12-27T22:08:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1649918976 bytes 2023-12-27T22:08:32.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_48 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:08:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:08:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2023-12-27T22:08:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:08:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-27T22:08:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:08:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-27T22:08:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:08:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-27T22:08:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_48': 2023-12-27T22:08:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:08:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14d08a5a61dc 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14d08a5a61dc 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:32 2023 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:32 2023 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:32 2023 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 526175a8-e0bc-47f4-b966-8b42df9ad7e6 2023-12-27T22:08:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:08:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:08:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-27T22:08:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:08:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:08:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T22:08:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:08:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:08:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 24269824 bytes 2023-12-27T22:08:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 74606592 bytes 2023-12-27T22:08:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 96779264 bytes 2023-12-27T22:08:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 166474752 bytes 2023-12-27T22:08:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 209709056 bytes 2023-12-27T22:08:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 277976064 bytes 2023-12-27T22:08:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 309743616 bytes 2023-12-27T22:08:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 369616896 bytes 2023-12-27T22:08:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 391495680 bytes 2023-12-27T22:08:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 428599296 bytes 2023-12-27T22:08:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 434503680 bytes 2023-12-27T22:08:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 449418240 bytes 2023-12-27T22:08:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 471045120 bytes 2023-12-27T22:08:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 515555328 bytes 2023-12-27T22:08:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 566752256 bytes 2023-12-27T22:08:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 654149632 bytes 2023-12-27T22:08:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 685620224 bytes 2023-12-27T22:08:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 735293440 bytes 2023-12-27T22:08:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 799003648 bytes 2023-12-27T22:08:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 859174912 bytes 2023-12-27T22:08:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 952273920 bytes 2023-12-27T22:08:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1017271296 bytes 2023-12-27T22:08:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1091546112 bytes 2023-12-27T22:08:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1096435712 bytes 2023-12-27T22:08:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1136456704 bytes 2023-12-27T22:08:39.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1215483904 bytes 2023-12-27T22:08:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1323655168 bytes 2023-12-27T22:08:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1413530624 bytes 2023-12-27T22:08:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1450172416 bytes 2023-12-27T22:08:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1459849216 bytes 2023-12-27T22:08:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1482563584 bytes 2023-12-27T22:08:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1531003904 bytes 2023-12-27T22:08:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1598605312 bytes 2023-12-27T22:08:41.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1630832640 bytes 2023-12-27T22:08:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1698745344 bytes 2023-12-27T22:08:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1722122240 bytes 2023-12-27T22:08:41.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1777571840 bytes 2023-12-27T22:08:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1800144896 bytes 2023-12-27T22:08:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1823619072 bytes 2023-12-27T22:08:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1830221824 bytes 2023-12-27T22:08:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1873497088 bytes 2023-12-27T22:08:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1937483776 bytes 2023-12-27T22:08:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1977588736 bytes 2023-12-27T22:08:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2031209472 bytes 2023-12-27T22:08:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2079333376 bytes 2023-12-27T22:08:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2139337728 bytes 2023-12-27T22:08:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2219932672 bytes 2023-12-27T22:08:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2265362432 bytes 2023-12-27T22:08:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2302239744 bytes 2023-12-27T22:08:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2376962048 bytes 2023-12-27T22:08:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2391026688 bytes 2023-12-27T22:08:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2413609984 bytes 2023-12-27T22:08:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2483738624 bytes 2023-12-27T22:08:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2484318208 bytes 2023-12-27T22:08:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2504419328 bytes 2023-12-27T22:08:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2537525248 bytes 2023-12-27T22:08:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2587074560 bytes 2023-12-27T22:08:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2621434880 bytes 2023-12-27T22:08:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2689852416 bytes 2023-12-27T22:08:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2747576320 bytes 2023-12-27T22:08:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2825712640 bytes 2023-12-27T22:08:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2826312704 bytes 2023-12-27T22:08:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2865163264 bytes 2023-12-27T22:08:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2928509952 bytes 2023-12-27T22:08:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2993195008 bytes 2023-12-27T22:08:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 3028773888 bytes 2023-12-27T22:08:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 3072250880 bytes 2023-12-27T22:08:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 3112833024 bytes 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_49 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-27T22:08:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:08:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:08:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:08:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:08:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2023-12-27T22:08:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:08:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-27T22:08:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:08:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-27T22:08:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:08:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-27T22:08:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_49': 2023-12-27T22:08:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:08:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14dcc663349c 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14dcc663349c 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:48 2023 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:48 2023 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:48 2023 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 13149630-0d06-4f7b-80ad-5c081ac5165b 2023-12-27T22:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:08:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:08:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-27T22:08:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:08:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:08:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=7 2023-12-27T22:08:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:08:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:08:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 2785280 bytes 2023-12-27T22:08:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 43665408 bytes 2023-12-27T22:08:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 111755264 bytes 2023-12-27T22:08:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 138071040 bytes 2023-12-27T22:08:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 208507904 bytes 2023-12-27T22:08:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 218575872 bytes 2023-12-27T22:08:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 282798080 bytes 2023-12-27T22:08:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 354640896 bytes 2023-12-27T22:08:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 417729536 bytes 2023-12-27T22:08:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 456358912 bytes 2023-12-27T22:08:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 508277760 bytes 2023-12-27T22:08:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 508900352 bytes 2023-12-27T22:08:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 592987136 bytes 2023-12-27T22:08:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 665963520 bytes 2023-12-27T22:08:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 681302016 bytes 2023-12-27T22:08:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 729091072 bytes 2023-12-27T22:08:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 738910208 bytes 2023-12-27T22:08:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 801926144 bytes 2023-12-27T22:08:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 817033216 bytes 2023-12-27T22:08:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 839870464 bytes 2023-12-27T22:08:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 850228224 bytes 2023-12-27T22:08:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 959894528 bytes 2023-12-27T22:08:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1016675328 bytes 2023-12-27T22:08:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1026566144 bytes 2023-12-27T22:08:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1054227456 bytes 2023-12-27T22:08:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1074442240 bytes 2023-12-27T22:08:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1127301120 bytes 2023-12-27T22:08:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1257638912 bytes 2023-12-27T22:08:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1268976640 bytes 2023-12-27T22:08:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1307723776 bytes 2023-12-27T22:08:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1331223552 bytes 2023-12-27T22:08:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1331846144 bytes 2023-12-27T22:08:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1392342016 bytes 2023-12-27T22:08:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1436709888 bytes 2023-12-27T22:08:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1462987776 bytes 2023-12-27T22:08:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:08:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:08:56.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_50 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n snapshot ']' 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-27T22:08:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 128M ']' 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=128M 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:08:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2023-12-27T22:08:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' image = image ']' 2023-12-27T22:08:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-27T22:08:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:08:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:08:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-27T22:08:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:08:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-27T22:08:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_50': 2023-12-27T22:08:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 14e83f6ea18e 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.14e83f6ea18e 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:56 2023 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:56 2023 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:56 2023 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 8f0ce147-0f9e-4455-a07c-cb25053ca356 2023-12-27T22:08:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:08:57.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n '' ']' 2023-12-27T22:08:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-27T22:08:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:08:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:08:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:08:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:08:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=14 2023-12-27T22:08:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:08:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:08:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 30545920 bytes 2023-12-27T22:08:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 31166464 bytes 2023-12-27T22:08:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 48849920 bytes 2023-12-27T22:08:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 103582720 bytes 2023-12-27T22:08:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 120830976 bytes 2023-12-27T22:08:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 153596928 bytes 2023-12-27T22:08:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 181153792 bytes 2023-12-27T22:08:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 267823104 bytes 2023-12-27T22:08:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 296396800 bytes 2023-12-27T22:08:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 365210624 bytes 2023-12-27T22:08:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 417026048 bytes 2023-12-27T22:08:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 445974528 bytes 2023-12-27T22:08:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 493921280 bytes 2023-12-27T22:08:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 564311040 bytes 2023-12-27T22:09:00.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 621973504 bytes 2023-12-27T22:09:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 719127552 bytes 2023-12-27T22:09:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 819990528 bytes 2023-12-27T22:09:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 890882048 bytes 2023-12-27T22:09:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 915704832 bytes 2023-12-27T22:09:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 916294656 bytes 2023-12-27T22:09:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 938515456 bytes 2023-12-27T22:09:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 969575424 bytes 2023-12-27T22:09:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1034107904 bytes 2023-12-27T22:09:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1071789056 bytes 2023-12-27T22:09:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1134946304 bytes 2023-12-27T22:09:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1192788992 bytes 2023-12-27T22:09:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1249648640 bytes 2023-12-27T22:09:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1278923776 bytes 2023-12-27T22:09:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1304262656 bytes 2023-12-27T22:09:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1317129216 bytes 2023-12-27T22:09:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1321592832 bytes 2023-12-27T22:09:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1396033536 bytes 2023-12-27T22:09:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1402156032 bytes 2023-12-27T22:09:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1448855552 bytes 2023-12-27T22:09:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1525512192 bytes 2023-12-27T22:09:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1600756736 bytes 2023-12-27T22:09:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1663091712 bytes 2023-12-27T22:09:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1750401024 bytes 2023-12-27T22:09:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1775396864 bytes 2023-12-27T22:09:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1864209408 bytes 2023-12-27T22:09:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1890612224 bytes 2023-12-27T22:09:06.181 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-27T22:09:06.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-27T22:09:06.181 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-27T22:09:06.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T22:09:06.182+0000 7fd106a6e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 59289) UID: 1000 2023-12-27T22:09:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 2012252160 bytes 2023-12-27T22:09:06.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:09:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 2084978688 bytes 2023-12-27T22:09:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 2104375296 bytes 2023-12-27T22:09:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2152444928 bytes 2023-12-27T22:09:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2170238976 bytes 2023-12-27T22:09:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2286548992 bytes 2023-12-27T22:09:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2331029504 bytes 2023-12-27T22:09:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2370115584 bytes 2023-12-27T22:09:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2407277568 bytes 2023-12-27T22:09:08.091 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7505933209210858) > weight (0.25) 2023-12-27T22:09:08.091 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-27T22:09:08.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-27T22:09:08.092 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T22:09:08.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi196.stderr:2023-12-27T22:09:08.092+0000 7fca857b7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 50570) UID: 1000 2023-12-27T22:09:08.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:09:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2498337792 bytes 2023-12-27T22:09:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2555444224 bytes 2023-12-27T22:09:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2573725696 bytes 2023-12-27T22:09:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2579241984 bytes 2023-12-27T22:09:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2618985472 bytes 2023-12-27T22:09:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2671211520 bytes 2023-12-27T22:09:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2733052928 bytes 2023-12-27T22:09:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2839729152 bytes 2023-12-27T22:09:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2861310976 bytes 2023-12-27T22:09:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2978862080 bytes 2023-12-27T22:09:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 3001481216 bytes 2023-12-27T22:09:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 3099677696 bytes 2023-12-27T22:09:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 3158976512 bytes 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_count=50 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local count 2023-12-27T22:09:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_count=0 2023-12-27T22:09:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:09:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 40 2023-12-27T22:09:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in `seq 1 40` 2023-12-27T22:09:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 1 -ne 1 2023-12-27T22:09:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-27T22:09:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep 'images: ' 2023-12-27T22:09:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cut '-d ' -f 2 2023-12-27T22:09:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T22:09:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ count=50 2023-12-27T22:09:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 50 = 50 2023-12-27T22:09:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-27T22:09:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state 2023-12-27T22:09:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 40 2023-12-27T22:09:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in `seq 1 40` 2023-12-27T22:09:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 1 -ne 1 2023-12-27T22:09:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-27T22:09:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep 'image health:' 2023-12-27T22:09:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cut '-d ' -f 3 2023-12-27T22:09:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=OK 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test OK = ERROR 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test OK = OK 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_count=50 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local count 2023-12-27T22:09:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_count=0 2023-12-27T22:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 40 2023-12-27T22:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in `seq 1 40` 2023-12-27T22:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 1 -ne 1 2023-12-27T22:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-27T22:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep 'images: ' 2023-12-27T22:09:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cut '-d ' -f 2 2023-12-27T22:09:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ count=50 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 50 = 50 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state 2023-12-27T22:09:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 40 2023-12-27T22:09:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in `seq 1 40` 2023-12-27T22:09:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 1 -ne 1 2023-12-27T22:09:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-27T22:09:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep 'image health:' 2023-12-27T22:09:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cut '-d ' -f 3 2023-12-27T22:09:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T22:09:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=WARNING 2023-12-27T22:09:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test WARNING = ERROR 2023-12-27T22:09:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test WARNING = OK 2023-12-27T22:09:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in `seq 1 40` 2023-12-27T22:09:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 2 -ne 1 2023-12-27T22:09:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 30 2023-12-27T22:09:12.092 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-27T22:09:12.093 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:09:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:09:12.093 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-27T22:09:12.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-27T22:09:12.094 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-27T22:09:12.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-27T22:09:12.098 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-12-27T22:09:12.182 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-27T22:09:12.182 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:09:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:09:12.182 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-27T22:09:12.182 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T22:09:12.183 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T22:09:12.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T22:09:12.185 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2023-12-27T22:09:25.116 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:09:25.114+0000 7fc355a18700 -1 reset not still connected to 0x55deeb2bdad0 2023-12-27T22:09:25.116 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:09:25.114+0000 7fc355a18700 -1 reset not still connected to 0x55deef867e10 2023-12-27T22:09:25.120 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:09:25.120+0000 7fc355a18700 -1 reset not still connected to 0x55deefba92b0 2023-12-27T22:09:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-27T22:09:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep 'image health:' 2023-12-27T22:09:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cut '-d ' -f 3 2023-12-27T22:09:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T22:09:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=OK 2023-12-27T22:09:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test OK = ERROR 2023-12-27T22:09:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test OK = OK 2023-12-27T22:09:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ testlog 'TEST: compare many images' 2023-12-27T22:09:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee -a /tmp/tmp.yuhsqMaECB/rbd-mirror.test.log 2023-12-27T22:09:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ date '+%F %T' 2023-12-27T22:09:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo 2023-12-27 22:09:41 TEST: compare many images 2023-12-27T22:09:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-27 22:09:41 TEST: compare many images 2023-12-27T22:09:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 50 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_1 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:09:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:09:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-27T22:09:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:09:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:09:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:09:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:09:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:09:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:09:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:09:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:09:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:09:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:09:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:09:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:09:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:09:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:09:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:09:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:09:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:09:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:09:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:09:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-27T22:09:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:09:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:09:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:09:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:09:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-27T22:09:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:09:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_1.status 2023-12-27T22:09:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_1.status 2023-12-27T22:09:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:09:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.status 2023-12-27T22:09:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_1.status 2023-12-27T22:09:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_1.status 2023-12-27T22:09:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:09:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-27T22:09:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-27T22:09:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 119 2023-12-27T22:09:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.status 2023-12-27T22:09:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.status 2023-12-27T22:09:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-27T22:09:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:09:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=119 2023-12-27T22:09:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:09:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:09:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-27T22:09:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=68 2023-12-27T22:09:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 119 = 68 2023-12-27T22:09:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:09:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-27T22:09:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=68 2023-12-27T22:09:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 119 = 68 2023-12-27T22:09:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:09:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-27T22:09:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:09:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:09:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:09:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:09:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:09:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 119 = '' 2023-12-27T22:09:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:46.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:09:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.status 2023-12-27T22:09:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-27T22:09:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=119 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 119 = 119 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:09:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:09:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:09:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-27T22:09:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_1': 2023-12-27T22:09:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:09:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:09:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:09:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b567f224d 2023-12-27T22:09:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b567f224d 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:11 2023 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:11 2023 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:11 2023 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 42426da1-e209-4023-8d5f-43ff971b1cd6 2023-12-27T22:09:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.export 2023-12-27T22:09:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.export 2023-12-27T22:09:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.export 2023-12-27T22:09:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.export 2023-12-27T22:09:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:09:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.export 2023-12-27T22:09:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:09:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.export 2023-12-27T22:09:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_1.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_1.export 2023-12-27T22:09:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_2 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:09:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:09:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-27T22:09:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-27T22:09:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:09:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:09:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:09:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:09:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:09:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:09:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:09:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:09:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:09:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:09:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:09:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:09:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:09:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:09:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:09:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:09:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:09:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:09:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:09:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-27T22:09:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:09:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:09:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:09:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:09:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:09:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_2.status 2023-12-27T22:09:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_2.status 2023-12-27T22:09:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.status 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_2.status 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_2.status 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-27T22:09:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 121 2023-12-27T22:09:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.status 2023-12-27T22:09:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.status 2023-12-27T22:09:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-27T22:09:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=121 2023-12-27T22:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:09:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-27T22:09:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=69 2023-12-27T22:09:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 121 = 69 2023-12-27T22:09:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:09:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-27T22:09:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=69 2023-12-27T22:09:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 121 = 69 2023-12-27T22:09:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:09:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-27T22:09:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:09:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:09:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:09:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:09:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:09:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 121 = '' 2023-12-27T22:09:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:09:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:09:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.status 2023-12-27T22:09:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-27T22:09:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:09:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:09:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=121 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 121 = 121 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:09:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:09:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-27T22:09:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_2': 2023-12-27T22:09:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:09:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:09:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:09:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b4e86e104 2023-12-27T22:09:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b4e86e104 2023-12-27T22:09:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:09:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:09:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:09:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:09:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:25 2023 2023-12-27T22:09:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:25 2023 2023-12-27T22:09:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:25 2023 2023-12-27T22:09:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:09:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:09:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:09:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: f4c094d4-f920-4b5f-ade9-72239f34d084 2023-12-27T22:09:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:09:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-27T22:09:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:09:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:09:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:09:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.export 2023-12-27T22:09:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.export 2023-12-27T22:09:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.export 2023-12-27T22:09:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.export 2023-12-27T22:09:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:09:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.export 2023-12-27T22:09:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:09:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.export 2023-12-27T22:09:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_2.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_2.export 2023-12-27T22:09:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:09:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:09:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_3 2023-12-27T22:09:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-27T22:09:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:09:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:09:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:09:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:09:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:09:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-27T22:10:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:10:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:10:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:10:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:10:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:10:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-27T22:10:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:10:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:10:00.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:10:00.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:10:00.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-27T22:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_3.status 2023-12-27T22:10:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_3.status 2023-12-27T22:10:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.status 2023-12-27T22:10:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_3.status 2023-12-27T22:10:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_3.status 2023-12-27T22:10:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-27T22:10:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-27T22:10:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 123 2023-12-27T22:10:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.status 2023-12-27T22:10:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.status 2023-12-27T22:10:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-27T22:10:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:10:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=123 2023-12-27T22:10:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:10:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-27T22:10:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=70 2023-12-27T22:10:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 123 = 70 2023-12-27T22:10:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:10:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-27T22:10:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=70 2023-12-27T22:10:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 123 = 70 2023-12-27T22:10:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:10:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-27T22:10:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 123 = '' 2023-12-27T22:10:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:10:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.status 2023-12-27T22:10:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-27T22:10:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=123 2023-12-27T22:10:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 123 = 123 2023-12-27T22:10:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-27T22:10:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:10:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-27T22:10:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_3': 2023-12-27T22:10:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:10:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:10:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:10:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bbd9eed52 2023-12-27T22:10:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bbd9eed52 2023-12-27T22:10:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:36 2023 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:36 2023 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:36 2023 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d21ceb08-f137-46c9-a16b-ae39e2b1220d 2023-12-27T22:10:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:10:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-27T22:10:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.export 2023-12-27T22:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.export 2023-12-27T22:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.export 2023-12-27T22:10:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.export 2023-12-27T22:10:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.export 2023-12-27T22:10:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.export 2023-12-27T22:10:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_3.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_3.export 2023-12-27T22:10:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:10:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_4 2023-12-27T22:10:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-27T22:10:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-27T22:10:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:10:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-27T22:10:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-27T22:10:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:10:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:10:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:10:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:10:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:10:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:10:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:10:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_4.status 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_4.status 2023-12-27T22:10:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.status 2023-12-27T22:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_4.status 2023-12-27T22:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_4.status 2023-12-27T22:10:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-27T22:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-27T22:10:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 125 2023-12-27T22:10:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.status 2023-12-27T22:10:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:10.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.status 2023-12-27T22:10:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-27T22:10:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:10:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=125 2023-12-27T22:10:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:10:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-27T22:10:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=71 2023-12-27T22:10:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 125 = 71 2023-12-27T22:10:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:10:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-27T22:10:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=71 2023-12-27T22:10:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 125 = 71 2023-12-27T22:10:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:10:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-27T22:10:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 125 = '' 2023-12-27T22:10:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:11.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:10:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.status 2023-12-27T22:10:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-27T22:10:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=125 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 125 = 125 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:10:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-27T22:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_4': 2023-12-27T22:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bc7848917 2023-12-27T22:10:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bc7848917 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:46 2023 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:46 2023 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:46 2023 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: c0d17036-eff0-4639-8b58-29f45ac3edc8 2023-12-27T22:10:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.export 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.export 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.export 2023-12-27T22:10:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.export 2023-12-27T22:10:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.export 2023-12-27T22:10:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.export 2023-12-27T22:10:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_4.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_4.export 2023-12-27T22:10:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_5 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:10:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-27T22:10:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-27T22:10:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:10:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:17.099 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T22:10:17.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T22:10:17.100 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T22:10:17.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:10:17.101+0000 7f24ac3f8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 58387) UID: 1000 2023-12-27T22:10:17.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:10:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-27T22:10:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:10:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:10:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:10:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-27T22:10:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-27T22:10:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:10:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:10:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:10:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:10:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-27T22:10:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:10:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:10:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:10:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-12-27T22:10:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2023-12-27T22:10:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T22:10:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T22:10:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T22:10:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:10:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:10:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:10:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2023-12-27T22:10:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 1 2023-12-27T22:10:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-27T22:10:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-27T22:10:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-27T22:10:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-27T22:10:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:10:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:10:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:10:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-27T22:10:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:10:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:10:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-12-27T22:10:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T22:10:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T22:10:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:10:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:10:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:10:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2023-12-27T22:10:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 2 2023-12-27T22:10:21.100 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T22:10:21.100 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:10:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:10:21.101 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T22:10:21.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T22:10:21.101 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T22:10:21.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T22:10:21.105 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-27T22:10:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-27T22:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-27T22:10:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:10:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:10:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:10:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-27T22:10:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-27T22:10:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:10:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:10:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:10:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-12-27T22:10:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:10:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:10:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:10:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:10:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:10:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:10:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:10:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_5.status 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_5.status 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.status 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_5.status 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_5.status 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.267+0000 7f8280ae9700 -1 reset not still connected to 0x55c8cc418c30 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8ccadc9c0 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8ccadd450 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8cd1f4270 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8cd1f4750 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8cd1f4ea0 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8cd1f5930 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8dbcb4270 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8dc1d05b0 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8dc1d1ba0 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8dc1d1ee0 2023-12-27T22:10:22.269 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8dc2dba00 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8dccd0680 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.268+0000 7f8280ae9700 -1 reset not still connected to 0x55c8ddb6c8f0 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.269+0000 7f8280ae9700 -1 reset not still connected to 0x55c8ddb6dc70 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.269+0000 7f8280ae9700 -1 reset not still connected to 0x55c8de795e10 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.269+0000 7f8280ae9700 -1 reset not still connected to 0x55c8deed4820 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.269+0000 7f8280ae9700 -1 reset not still connected to 0x55c8deed5380 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.269+0000 7f8280ae9700 -1 reset not still connected to 0x55c8deed56c0 2023-12-27T22:10:22.270 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:10:22.269+0000 7f8280ae9700 -1 reset not still connected to 0x55c8deed5790 2023-12-27T22:10:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 127 2023-12-27T22:10:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.status 2023-12-27T22:10:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.status 2023-12-27T22:10:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-27T22:10:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:10:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=127 2023-12-27T22:10:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:10:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-27T22:10:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=72 2023-12-27T22:10:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 127 = 72 2023-12-27T22:10:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-27T22:10:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=72 2023-12-27T22:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 127 = 72 2023-12-27T22:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-27T22:10:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 127 = '' 2023-12-27T22:10:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:10:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.status 2023-12-27T22:10:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-27T22:10:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=127 2023-12-27T22:10:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 127 = 127 2023-12-27T22:10:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-27T22:10:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:10:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_5': 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b31cab387 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b31cab387 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:10:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:00:55 2023 2023-12-27T22:10:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:00:55 2023 2023-12-27T22:10:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:00:55 2023 2023-12-27T22:10:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:10:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:10:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:10:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d2ad7245-32f6-4c55-975e-2cc6979edf3e 2023-12-27T22:10:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:10:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-27T22:10:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:10:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:10:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.export 2023-12-27T22:10:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.export 2023-12-27T22:10:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.export 2023-12-27T22:10:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.export 2023-12-27T22:10:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.export 2023-12-27T22:10:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.export 2023-12-27T22:10:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_5.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_5.export 2023-12-27T22:10:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:10:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_6 2023-12-27T22:10:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-27T22:10:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:10:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-27T22:10:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-27T22:10:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:10:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:10:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-27T22:10:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-27T22:10:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-27T22:10:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:10:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:10:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:10:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-12-27T22:10:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-12-27T22:10:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:10:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:10:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:10:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2023-12-27T22:10:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:10:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:10:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:10:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:10:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:10:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:10:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:10:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-27T22:10:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_6.status 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_6.status 2023-12-27T22:10:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.status 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_6.status 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_6.status 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-27T22:10:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 129 2023-12-27T22:10:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.status 2023-12-27T22:10:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.status 2023-12-27T22:10:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-27T22:10:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:10:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=129 2023-12-27T22:10:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:10:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-27T22:10:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=73 2023-12-27T22:10:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 129 = 73 2023-12-27T22:10:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:10:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-27T22:10:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=73 2023-12-27T22:10:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 129 = 73 2023-12-27T22:10:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:10:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-27T22:10:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 129 = '' 2023-12-27T22:10:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:10:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.status 2023-12-27T22:10:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-27T22:10:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=129 2023-12-27T22:10:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 129 = 129 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:10:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_6': 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b1c07425a 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b1c07425a 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:10:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:03 2023 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:03 2023 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:03 2023 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 7b6aaf8f-abec-4e77-a74c-774b6db907b0 2023-12-27T22:10:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:10:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-27T22:10:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:10:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:10:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.export 2023-12-27T22:10:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.export 2023-12-27T22:10:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.export 2023-12-27T22:10:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.export 2023-12-27T22:10:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.export 2023-12-27T22:10:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.export 2023-12-27T22:10:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_6.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_6.export 2023-12-27T22:10:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_7 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:10:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-27T22:10:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-27T22:10:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:10:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-27T22:10:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-27T22:10:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:10:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:10:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:10:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-27T22:10:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-27T22:10:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-12-27T22:10:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:10:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:10:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-12-27T22:10:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:10:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:10:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:10:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:10:37.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_7.status 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_7.status 2023-12-27T22:10:37.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.status 2023-12-27T22:10:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_7.status 2023-12-27T22:10:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_7.status 2023-12-27T22:10:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-27T22:10:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-27T22:10:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 131 2023-12-27T22:10:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.status 2023-12-27T22:10:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.status 2023-12-27T22:10:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-27T22:10:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:10:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=131 2023-12-27T22:10:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:10:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-27T22:10:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=74 2023-12-27T22:10:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 131 = 74 2023-12-27T22:10:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:10:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-27T22:10:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=74 2023-12-27T22:10:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 131 = 74 2023-12-27T22:10:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:10:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-27T22:10:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 131 = '' 2023-12-27T22:10:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:10:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.status 2023-12-27T22:10:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-27T22:10:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=131 2023-12-27T22:10:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 131 = 131 2023-12-27T22:10:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-27T22:10:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:10:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-27T22:10:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_7': 2023-12-27T22:10:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:10:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:10:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:10:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b2ff9333b 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b2ff9333b 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:15 2023 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:15 2023 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:15 2023 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 5a81880e-ca19-4e3d-9afa-fac4d233817b 2023-12-27T22:10:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.export 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.export 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.export 2023-12-27T22:10:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.export 2023-12-27T22:10:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.export 2023-12-27T22:10:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.export 2023-12-27T22:10:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_7.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_7.export 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_8 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:10:43.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.40372054454990447) > weight (0.25) 2023-12-27T22:10:43.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8590418947824922) > weight (0.25) 2023-12-27T22:10:43.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.29125448751541294) > weight (0.25) 2023-12-27T22:10:43.186 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7423724127702312) > weight (0.25) 2023-12-27T22:10:43.186 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-12-27T22:10:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-27T22:10:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-27T22:10:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:10:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:10:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:10:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-27T22:10:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:10:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:10:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:10:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:10:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:10:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:10:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-27T22:10:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-27T22:10:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-27T22:10:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:10:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:10:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:10:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:10:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2023-12-27T22:10:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-27T22:10:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2023-12-27T22:10:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:10:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:10:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:10:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2023-12-27T22:10:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:10:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:10:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:10:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:10:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:10:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:10:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:10:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_8.status 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_8.status 2023-12-27T22:10:45.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.status 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_8.status 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_8.status 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-27T22:10:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 133 2023-12-27T22:10:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.status 2023-12-27T22:10:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.status 2023-12-27T22:10:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-27T22:10:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:10:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=133 2023-12-27T22:10:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:10:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-27T22:10:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=75 2023-12-27T22:10:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 133 = 75 2023-12-27T22:10:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:10:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-27T22:10:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 133 = '' 2023-12-27T22:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:10:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-27T22:10:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 133 = '' 2023-12-27T22:10:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:10:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.status 2023-12-27T22:10:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-27T22:10:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=133 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 133 = 133 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:10:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-27T22:10:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_8': 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bc65e07cf 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bc65e07cf 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:29 2023 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:29 2023 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:29 2023 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:10:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:10:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d4452c8f-ad61-428d-a4d6-8e781d2d9950 2023-12-27T22:10:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:10:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-27T22:10:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:10:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:10:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.export 2023-12-27T22:10:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.export 2023-12-27T22:10:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.export 2023-12-27T22:10:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.export 2023-12-27T22:10:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.export 2023-12-27T22:10:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.export 2023-12-27T22:10:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_8.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_8.export 2023-12-27T22:10:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:10:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_9 2023-12-27T22:10:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-27T22:10:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:10:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-27T22:10:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:10:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:10:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:10:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:10:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:10:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:10:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:10:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:10:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-27T22:10:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-27T22:10:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:10:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:10:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:10:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:10:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:10:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:10:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-12-27T22:10:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-27T22:10:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:10:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:10:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:10:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:10:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:10:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:10:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2023-12-27T22:10:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:10:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:10:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:10:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:10:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:10:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:10:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:10:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_9.status 2023-12-27T22:10:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_9.status 2023-12-27T22:10:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.status 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_9.status 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_9.status 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-27T22:10:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 135 2023-12-27T22:10:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.status 2023-12-27T22:10:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:10:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.status 2023-12-27T22:10:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-27T22:10:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:10:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=135 2023-12-27T22:10:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:10:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-27T22:10:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=76 2023-12-27T22:10:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 135 = 76 2023-12-27T22:10:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:10:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-27T22:10:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=76 2023-12-27T22:10:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 135 = 76 2023-12-27T22:10:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:10:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-27T22:10:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:10:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:10:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:10:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:10:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:10:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 135 = '' 2023-12-27T22:10:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:10:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:10:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.status 2023-12-27T22:10:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-27T22:10:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:10:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:10:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=135 2023-12-27T22:10:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 135 = 135 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:10:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:10:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:10:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_9': 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b4eafd962 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b4eafd962 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:10:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:10:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:10:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:10:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:44 2023 2023-12-27T22:10:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:44 2023 2023-12-27T22:10:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:44 2023 2023-12-27T22:10:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:10:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:10:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:10:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: bf528422-e47f-45b4-ada3-2e77c12c15d7 2023-12-27T22:10:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.export 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.export 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.export 2023-12-27T22:10:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.export 2023-12-27T22:10:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:10:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.export 2023-12-27T22:11:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.export 2023-12-27T22:11:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_9.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_9.export 2023-12-27T22:11:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_10 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-27T22:11:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-27T22:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-27T22:11:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-27T22:11:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-27T22:11:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-27T22:11:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-27T22:11:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-12-27T22:11:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-12-27T22:11:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_10.status 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_10.status 2023-12-27T22:11:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.status 2023-12-27T22:11:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_10.status 2023-12-27T22:11:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_10.status 2023-12-27T22:11:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-27T22:11:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-27T22:11:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 137 2023-12-27T22:11:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.status 2023-12-27T22:11:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.status 2023-12-27T22:11:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-27T22:11:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=137 2023-12-27T22:11:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:11:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-27T22:11:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=77 2023-12-27T22:11:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 137 = 77 2023-12-27T22:11:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:11:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-27T22:11:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 137 = '' 2023-12-27T22:11:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:11:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-27T22:11:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 137 = '' 2023-12-27T22:11:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:11:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.status 2023-12-27T22:11:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-27T22:11:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=137 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 137 = 137 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:11:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-27T22:11:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_10': 2023-12-27T22:11:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b5cd8dc6a 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b5cd8dc6a 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:01:57 2023 2023-12-27T22:11:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:01:57 2023 2023-12-27T22:11:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:01:57 2023 2023-12-27T22:11:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:11:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:11:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:11:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: ffe88c0d-d6ed-4f88-83b3-e604b69fe96e 2023-12-27T22:11:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:11:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-27T22:11:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:11:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:11:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.export 2023-12-27T22:11:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.export 2023-12-27T22:11:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.export 2023-12-27T22:11:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.export 2023-12-27T22:11:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.export 2023-12-27T22:11:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.export 2023-12-27T22:11:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_10.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_10.export 2023-12-27T22:11:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_11 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-27T22:11:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-27T22:11:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-27T22:11:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-27T22:11:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-27T22:11:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-27T22:11:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-27T22:11:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-12-27T22:11:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-12-27T22:11:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_11.status 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_11.status 2023-12-27T22:11:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.status 2023-12-27T22:11:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_11.status 2023-12-27T22:11:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_11.status 2023-12-27T22:11:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-27T22:11:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-27T22:11:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 139 2023-12-27T22:11:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.status 2023-12-27T22:11:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.status 2023-12-27T22:11:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-27T22:11:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=139 2023-12-27T22:11:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:11:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-27T22:11:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=78 2023-12-27T22:11:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 139 = 78 2023-12-27T22:11:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:11:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-27T22:11:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 139 = '' 2023-12-27T22:11:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:11:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-27T22:11:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:13.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 139 = '' 2023-12-27T22:11:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:11:13.508 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:11:13.508+0000 7fa0daabc700 -1 reset not still connected to 0x55670d723860 2023-12-27T22:11:13.508 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:11:13.508+0000 7fa0daabc700 -1 reset not still connected to 0x55670db468f0 2023-12-27T22:11:13.508 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:11:13.508+0000 7fa0daabc700 -1 reset not still connected to 0x55670db46f70 2023-12-27T22:11:13.508 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:11:13.508+0000 7fa0daabc700 -1 reset not still connected to 0x55670dc645b0 2023-12-27T22:11:13.509 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:11:13.509+0000 7fa0daabc700 -1 reset not still connected to 0x55670e0224e0 2023-12-27T22:11:13.509 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:11:13.509+0000 7fa0daabc700 -1 reset not still connected to 0x55670e0229c0 2023-12-27T22:11:13.509 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:11:13.509+0000 7fa0daabc700 -1 reset not still connected to 0x55670e116000 2023-12-27T22:11:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.status 2023-12-27T22:11:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-27T22:11:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=139 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 139 = 139 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:11:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_11': 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b143025c 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b143025c 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:11:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:10 2023 2023-12-27T22:11:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:10 2023 2023-12-27T22:11:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:10 2023 2023-12-27T22:11:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:11:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:11:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:11:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 077dd2f0-f421-4104-9242-d6588dfb934d 2023-12-27T22:11:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:11:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-27T22:11:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:11:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:11:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.export 2023-12-27T22:11:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.export 2023-12-27T22:11:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.export 2023-12-27T22:11:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.export 2023-12-27T22:11:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.export 2023-12-27T22:11:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.export 2023-12-27T22:11:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_11.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_11.export 2023-12-27T22:11:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_12 2023-12-27T22:11:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-27T22:11:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-27T22:11:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-27T22:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-27T22:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-27T22:11:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-27T22:11:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-27T22:11:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-27T22:11:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-27T22:11:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-27T22:11:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_12.status 2023-12-27T22:11:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_12.status 2023-12-27T22:11:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.status 2023-12-27T22:11:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_12.status 2023-12-27T22:11:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_12.status 2023-12-27T22:11:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-27T22:11:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-27T22:11:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 141 2023-12-27T22:11:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.status 2023-12-27T22:11:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.status 2023-12-27T22:11:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-27T22:11:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=141 2023-12-27T22:11:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:11:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-27T22:11:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=79 2023-12-27T22:11:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 141 = 79 2023-12-27T22:11:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:11:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-27T22:11:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 141 = '' 2023-12-27T22:11:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:11:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-27T22:11:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 141 = '' 2023-12-27T22:11:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.status 2023-12-27T22:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-27T22:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=141 2023-12-27T22:11:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 141 = 141 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-27T22:11:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_12': 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bd390f3ac 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bd390f3ac 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:18 2023 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:18 2023 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:18 2023 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:11:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:11:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:11:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: a041afaf-4f1b-4932-be77-439bad206770 2023-12-27T22:11:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:11:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-27T22:11:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:11:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:11:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.export 2023-12-27T22:11:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.export 2023-12-27T22:11:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.export 2023-12-27T22:11:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.export 2023-12-27T22:11:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.export 2023-12-27T22:11:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.export 2023-12-27T22:11:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_12.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_12.export 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_13 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-27T22:11:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-27T22:11:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-27T22:11:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-12-27T22:11:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-27T22:11:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2023-12-27T22:11:27.106 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-27T22:11:27.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-27T22:11:27.106 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-27T22:11:27.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:11:27.108+0000 7f7108a10700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 63266) UID: 1000 2023-12-27T22:11:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-27T22:11:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_13.status 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_13.status 2023-12-27T22:11:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.status 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_13.status 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_13.status 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-27T22:11:27.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:11:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 143 2023-12-27T22:11:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.status 2023-12-27T22:11:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.status 2023-12-27T22:11:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-27T22:11:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=143 2023-12-27T22:11:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:11:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-27T22:11:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=80 2023-12-27T22:11:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 143 = 80 2023-12-27T22:11:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:11:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-27T22:11:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 143 = '' 2023-12-27T22:11:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:11:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-27T22:11:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 143 = '' 2023-12-27T22:11:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:11:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.status 2023-12-27T22:11:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-27T22:11:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=143 2023-12-27T22:11:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 143 = 143 2023-12-27T22:11:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-27T22:11:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:11:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-27T22:11:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_13': 2023-12-27T22:11:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:11:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:11:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b2e1c61eb 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b2e1c61eb 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:30 2023 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:30 2023 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:30 2023 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: f7d5f611-7612-48c4-a55c-a9116d980d1b 2023-12-27T22:11:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.export 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.export 2023-12-27T22:11:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.export 2023-12-27T22:11:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.export 2023-12-27T22:11:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.export 2023-12-27T22:11:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.export 2023-12-27T22:11:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_13.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_13.export 2023-12-27T22:11:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_14 2023-12-27T22:11:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-27T22:11:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-27T22:11:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-27T22:11:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-27T22:11:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-27T22:11:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-27T22:11:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-27T22:11:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-27T22:11:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-27T22:11:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-27T22:11:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-12-27T22:11:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_14.status 2023-12-27T22:11:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_14.status 2023-12-27T22:11:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.status 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_14.status 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_14.status 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-27T22:11:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 145 2023-12-27T22:11:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.status 2023-12-27T22:11:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.status 2023-12-27T22:11:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-27T22:11:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=145 2023-12-27T22:11:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:11:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-27T22:11:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:36.107 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-27T22:11:36.107 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:11:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:11:36.107 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-27T22:11:36.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-27T22:11:36.108 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-27T22:11:36.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-27T22:11:36.111 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-27T22:11:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=81 2023-12-27T22:11:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 145 = 81 2023-12-27T22:11:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:11:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-27T22:11:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 145 = '' 2023-12-27T22:11:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:11:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-27T22:11:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 145 = '' 2023-12-27T22:11:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:11:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.status 2023-12-27T22:11:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-27T22:11:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=145 2023-12-27T22:11:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 145 = 145 2023-12-27T22:11:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-27T22:11:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:11:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-27T22:11:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_14': 2023-12-27T22:11:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:11:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:11:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:11:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b4d0711ce 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b4d0711ce 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:38 2023 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:38 2023 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:38 2023 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: cf062d67-4032-4099-9b25-373a590d58f7 2023-12-27T22:11:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.export 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.export 2023-12-27T22:11:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.export 2023-12-27T22:11:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.export 2023-12-27T22:11:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.export 2023-12-27T22:11:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.export 2023-12-27T22:11:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_14.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_14.export 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_15 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-27T22:11:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-27T22:11:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-27T22:11:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-27T22:11:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-27T22:11:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-27T22:11:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-12-27T22:11:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-12-27T22:11:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-27T22:11:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2023-12-27T22:11:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:43.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-27T22:11:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_15.status 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_15.status 2023-12-27T22:11:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.status 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_15.status 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_15.status 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-27T22:11:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 147 2023-12-27T22:11:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.status 2023-12-27T22:11:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.status 2023-12-27T22:11:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-27T22:11:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=147 2023-12-27T22:11:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:11:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-27T22:11:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=82 2023-12-27T22:11:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 147 = 82 2023-12-27T22:11:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:11:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-27T22:11:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 147 = '' 2023-12-27T22:11:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:11:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-27T22:11:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 147 = '' 2023-12-27T22:11:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:11:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.status 2023-12-27T22:11:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-27T22:11:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=147 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 147 = 147 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:11:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_15': 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109baa0a6d63 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109baa0a6d63 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:02:53 2023 2023-12-27T22:11:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:02:53 2023 2023-12-27T22:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:02:53 2023 2023-12-27T22:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 6b7b2978-4df8-43a6-826f-2914e643a2d2 2023-12-27T22:11:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-27T22:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:11:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.export 2023-12-27T22:11:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.export 2023-12-27T22:11:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.export 2023-12-27T22:11:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.export 2023-12-27T22:11:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.export 2023-12-27T22:11:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.export 2023-12-27T22:11:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_15.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_15.export 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_16 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-27T22:11:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-27T22:11:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-27T22:11:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-12-27T22:11:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-27T22:11:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2023-12-27T22:11:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_16.status 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_16.status 2023-12-27T22:11:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.status 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_16.status 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_16.status 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-27T22:11:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 149 2023-12-27T22:11:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.status 2023-12-27T22:11:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.status 2023-12-27T22:11:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-27T22:11:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=149 2023-12-27T22:11:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:11:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-27T22:11:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 149 = '' 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:11:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-27T22:11:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 149 = '' 2023-12-27T22:11:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:11:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-27T22:11:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:11:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:11:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:11:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:11:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 149 = '' 2023-12-27T22:11:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:11:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.status 2023-12-27T22:11:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-27T22:11:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:11:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=149 2023-12-27T22:11:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 149 = 149 2023-12-27T22:11:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-27T22:11:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:11:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:55.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7296453806791298) > weight (0.25) 2023-12-27T22:11:55.187 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.876485565808447) > weight (0.25) 2023-12-27T22:11:55.187 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-27T22:11:55.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-27T22:11:55.187 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T22:11:55.188 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr:2023-12-27T22:11:55.188+0000 7f09fd4bd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46570) UID: 1000 2023-12-27T22:11:55.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:11:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_16': 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b1ac6618e 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b1ac6618e 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:04 2023 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:04 2023 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:04 2023 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 299a19a7-69a7-45b4-804c-649ac34275ff 2023-12-27T22:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-27T22:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:11:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.export 2023-12-27T22:11:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.export 2023-12-27T22:11:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.export 2023-12-27T22:11:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.export 2023-12-27T22:11:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.export 2023-12-27T22:11:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:11:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.export 2023-12-27T22:11:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_16.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_16.export 2023-12-27T22:11:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_17 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:11:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-27T22:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:11:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-27T22:11:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:11:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:11:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:11:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:11:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-27T22:11:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:11:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:11:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:11:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:11:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:11:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:11:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:11:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:11:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-12-27T22:11:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-27T22:11:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:11:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:11:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:11:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:11:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:11:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:11:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:11:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:11:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:11:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2023-12-27T22:11:59.188 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-27T22:11:59.188 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:11:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:11:59.189 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-27T22:11:59.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-27T22:11:59.189 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-27T22:11:59.191 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-27T22:11:59.191 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-12-27T22:11:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:11:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:11:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:11:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:11:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_17.status 2023-12-27T22:11:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_17.status 2023-12-27T22:11:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.status 2023-12-27T22:11:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_17.status 2023-12-27T22:11:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_17.status 2023-12-27T22:11:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:11:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:11:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-27T22:11:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-27T22:11:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 151 2023-12-27T22:11:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.status 2023-12-27T22:11:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:11:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:11:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:11:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.status 2023-12-27T22:11:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-27T22:11:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:11:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:11:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=151 2023-12-27T22:11:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:11:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:11:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:12:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-27T22:12:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=84 2023-12-27T22:12:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 151 = 84 2023-12-27T22:12:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:12:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-27T22:12:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 151 = '' 2023-12-27T22:12:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:12:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:12:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-27T22:12:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 151 = '' 2023-12-27T22:12:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.status 2023-12-27T22:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-27T22:12:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=151 2023-12-27T22:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 151 = 151 2023-12-27T22:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-27T22:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:12:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:12:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-27T22:12:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_17': 2023-12-27T22:12:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:12:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:12:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:12:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bbf76eddb 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bbf76eddb 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:19 2023 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:19 2023 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:19 2023 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: fd00cd65-5352-4569-8fe7-a9ebb5daa8a3 2023-12-27T22:12:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.export 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.export 2023-12-27T22:12:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.export 2023-12-27T22:12:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.export 2023-12-27T22:12:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.export 2023-12-27T22:12:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.export 2023-12-27T22:12:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_17.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_17.export 2023-12-27T22:12:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_18 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:12:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-27T22:12:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-27T22:12:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:12:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:12:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:12:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-27T22:12:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:12:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:12:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:12:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:12:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:12:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:12:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:12:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:12:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-27T22:12:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:12:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:12:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:12:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-27T22:12:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-27T22:12:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:12:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:12:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:12:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-27T22:12:07.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-27T22:12:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-12-27T22:12:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:12:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:12:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:12:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-12-27T22:12:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:12:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:12:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:12:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:12:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_18.status 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_18.status 2023-12-27T22:12:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.status 2023-12-27T22:12:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_18.status 2023-12-27T22:12:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_18.status 2023-12-27T22:12:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-27T22:12:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-27T22:12:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 153 2023-12-27T22:12:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.status 2023-12-27T22:12:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.status 2023-12-27T22:12:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-27T22:12:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:12:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=153 2023-12-27T22:12:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:08.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-27T22:12:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=85 2023-12-27T22:12:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 153 = 85 2023-12-27T22:12:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-27T22:12:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 153 = '' 2023-12-27T22:12:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:12:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-27T22:12:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 153 = '' 2023-12-27T22:12:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:12:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.status 2023-12-27T22:12:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-27T22:12:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=153 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 153 = 153 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:12:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_18': 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109beb5af28 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109beb5af28 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:31 2023 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:31 2023 2023-12-27T22:12:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:31 2023 2023-12-27T22:12:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:12:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:12:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:12:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: e11e4f17-4382-4319-a42a-59efce51e962 2023-12-27T22:12:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:12:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-27T22:12:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:12:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:12:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.export 2023-12-27T22:12:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.export 2023-12-27T22:12:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.export 2023-12-27T22:12:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.export 2023-12-27T22:12:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.export 2023-12-27T22:12:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.export 2023-12-27T22:12:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_18.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_18.export 2023-12-27T22:12:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:12:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_19 2023-12-27T22:12:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-27T22:12:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:12:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:12:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:12:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:12:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:12:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:12:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-27T22:12:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-27T22:12:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:12:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:12:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:12:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-27T22:12:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-27T22:12:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2023-12-27T22:12:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:12:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:12:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:12:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:12:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:12:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:12:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:12:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:12:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2023-12-27T22:12:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:12:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:12:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:12:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:12:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:12:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_19.status 2023-12-27T22:12:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_19.status 2023-12-27T22:12:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.status 2023-12-27T22:12:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_19.status 2023-12-27T22:12:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_19.status 2023-12-27T22:12:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-27T22:12:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-27T22:12:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 155 2023-12-27T22:12:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.status 2023-12-27T22:12:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.status 2023-12-27T22:12:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-27T22:12:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:12:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=155 2023-12-27T22:12:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-27T22:12:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 155 = '' 2023-12-27T22:12:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-27T22:12:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 155 = '' 2023-12-27T22:12:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:12:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-27T22:12:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 155 = '' 2023-12-27T22:12:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:12:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.status 2023-12-27T22:12:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-27T22:12:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=155 2023-12-27T22:12:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 155 = 155 2023-12-27T22:12:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-27T22:12:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:12:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-27T22:12:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_19': 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bfc1d003c 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bfc1d003c 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:42 2023 2023-12-27T22:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:42 2023 2023-12-27T22:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:42 2023 2023-12-27T22:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: f3df9bf1-fabf-4121-bdef-2b2d3451c060 2023-12-27T22:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:12:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-27T22:12:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:12:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:12:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.export 2023-12-27T22:12:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.export 2023-12-27T22:12:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.export 2023-12-27T22:12:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.export 2023-12-27T22:12:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.export 2023-12-27T22:12:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.export 2023-12-27T22:12:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_19.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_19.export 2023-12-27T22:12:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:12:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_20 2023-12-27T22:12:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-27T22:12:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-27T22:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-27T22:12:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:12:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:12:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:12:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-27T22:12:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:12:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:12:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:12:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-27T22:12:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:12:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:12:23.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:12:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:12:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:12:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-27T22:12:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-27T22:12:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-27T22:12:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:12:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:12:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:12:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:12:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:12:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-27T22:12:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-27T22:12:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-27T22:12:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:12:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:12:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:12:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:12:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-27T22:12:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:12:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:12:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:12:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:12:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:12:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:12:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:12:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-27T22:12:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_20.status 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_20.status 2023-12-27T22:12:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.status 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_20.status 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_20.status 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-27T22:12:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 157 2023-12-27T22:12:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.status 2023-12-27T22:12:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.status 2023-12-27T22:12:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-27T22:12:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:12:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=157 2023-12-27T22:12:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-27T22:12:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=87 2023-12-27T22:12:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 157 = 87 2023-12-27T22:12:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-27T22:12:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 157 = '' 2023-12-27T22:12:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:12:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-27T22:12:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 157 = '' 2023-12-27T22:12:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:12:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.status 2023-12-27T22:12:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-27T22:12:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=157 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 157 = 157 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:12:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-27T22:12:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_20': 2023-12-27T22:12:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:12:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:12:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:12:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b4ef8e993 2023-12-27T22:12:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b4ef8e993 2023-12-27T22:12:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:03:52 2023 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:03:52 2023 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:03:52 2023 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 7149049a-7770-48be-805f-bc4a325f8c4b 2023-12-27T22:12:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:12:29.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.export 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.export 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.export 2023-12-27T22:12:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.export 2023-12-27T22:12:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.export 2023-12-27T22:12:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.export 2023-12-27T22:12:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_20.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_20.export 2023-12-27T22:12:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:12:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_21 2023-12-27T22:12:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-27T22:12:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:12:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-27T22:12:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-27T22:12:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:12:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-27T22:12:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:12:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:12:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:12:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-27T22:12:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:12:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:12:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:12:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:12:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:12:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:12:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:12:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:12:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:12:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:12:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:12:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-27T22:12:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-27T22:12:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:31.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:31.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:12:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:12:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:12:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-12-27T22:12:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-27T22:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:12:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:12:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:12:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-12-27T22:12:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:12:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:12:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:12:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:12:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:12:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:12:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:12:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-27T22:12:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_21.status 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_21.status 2023-12-27T22:12:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.status 2023-12-27T22:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_21.status 2023-12-27T22:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_21.status 2023-12-27T22:12:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-27T22:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-27T22:12:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 159 2023-12-27T22:12:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.status 2023-12-27T22:12:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.status 2023-12-27T22:12:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-27T22:12:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:12:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=159 2023-12-27T22:12:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-27T22:12:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=88 2023-12-27T22:12:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 159 = 88 2023-12-27T22:12:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-27T22:12:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 159 = '' 2023-12-27T22:12:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:12:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-27T22:12:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 159 = '' 2023-12-27T22:12:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:12:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.status 2023-12-27T22:12:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-27T22:12:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=159 2023-12-27T22:12:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 159 = 159 2023-12-27T22:12:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-27T22:12:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:12:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-27T22:12:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_21': 2023-12-27T22:12:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:12:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b41aa9bf3 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b41aa9bf3 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:12:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:04 2023 2023-12-27T22:12:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:04 2023 2023-12-27T22:12:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:04 2023 2023-12-27T22:12:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:12:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:12:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:12:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 06d7ea5e-0a1f-42f1-8ff9-fb3499537169 2023-12-27T22:12:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:12:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.export 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.export 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.export 2023-12-27T22:12:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.export 2023-12-27T22:12:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.export 2023-12-27T22:12:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.export 2023-12-27T22:12:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_21.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_21.export 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_22 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:12:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-27T22:12:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:12:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:12:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:12:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-27T22:12:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:12:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:12:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:12:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:12:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:12:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:12:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:12:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:12:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:12:40.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:12:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-27T22:12:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-27T22:12:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:40.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-27T22:12:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:12:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:12:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:12:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:12:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:12:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-27T22:12:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-27T22:12:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:12:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:12:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:12:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2023-12-27T22:12:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:12:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:12:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:12:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:12:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:12:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_22.status 2023-12-27T22:12:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_22.status 2023-12-27T22:12:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.status 2023-12-27T22:12:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_22.status 2023-12-27T22:12:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_22.status 2023-12-27T22:12:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-27T22:12:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:41.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-27T22:12:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 161 2023-12-27T22:12:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.status 2023-12-27T22:12:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.status 2023-12-27T22:12:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-27T22:12:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:12:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=161 2023-12-27T22:12:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-27T22:12:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=89 2023-12-27T22:12:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 161 = 89 2023-12-27T22:12:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-27T22:12:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:42.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 161 = '' 2023-12-27T22:12:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:12:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-27T22:12:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 161 = '' 2023-12-27T22:12:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:12:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.status 2023-12-27T22:12:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-27T22:12:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=161 2023-12-27T22:12:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 161 = 161 2023-12-27T22:12:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-27T22:12:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:12:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:45.111 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7963805212217937) > weight (0.25) 2023-12-27T22:12:45.112 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5938417087416277) > weight (0.25) 2023-12-27T22:12:45.112 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6041580673944529) > weight (0.25) 2023-12-27T22:12:45.112 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6525787473899812) > weight (0.25) 2023-12-27T22:12:45.112 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-12-27T22:12:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-27T22:12:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_22': 2023-12-27T22:12:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:12:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:12:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:12:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bd4f23965 2023-12-27T22:12:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bd4f23965 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:15 2023 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:15 2023 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:15 2023 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 18ab07a2-986f-4749-9454-ce1d87f8014f 2023-12-27T22:12:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.export 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.export 2023-12-27T22:12:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.export 2023-12-27T22:12:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.export 2023-12-27T22:12:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.export 2023-12-27T22:12:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.export 2023-12-27T22:12:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_22.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_22.export 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_23 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:12:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-27T22:12:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-27T22:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:12:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:12:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:12:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:12:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:12:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:12:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-27T22:12:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-27T22:12:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:12:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:12:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:12:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-27T22:12:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-27T22:12:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-12-27T22:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:12:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-12-27T22:12:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:12:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:12:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:12:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:12:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-27T22:12:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_23.status 2023-12-27T22:12:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_23.status 2023-12-27T22:12:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.status 2023-12-27T22:12:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_23.status 2023-12-27T22:12:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_23.status 2023-12-27T22:12:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-27T22:12:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-27T22:12:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 163 2023-12-27T22:12:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.status 2023-12-27T22:12:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.status 2023-12-27T22:12:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-27T22:12:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:12:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=163 2023-12-27T22:12:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-27T22:12:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=90 2023-12-27T22:12:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 163 = 90 2023-12-27T22:12:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-27T22:12:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 163 = '' 2023-12-27T22:12:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:12:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-27T22:12:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 163 = '' 2023-12-27T22:12:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:12:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.status 2023-12-27T22:12:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-27T22:12:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=163 2023-12-27T22:12:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 163 = 163 2023-12-27T22:12:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:12:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-27T22:12:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_23': 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b235d9559 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b235d9559 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:28 2023 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:28 2023 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:28 2023 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:12:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:12:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: a37ade65-cdd8-4035-b15e-da4bd79e5619 2023-12-27T22:12:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:12:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-27T22:12:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:12:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:12:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.export 2023-12-27T22:12:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.export 2023-12-27T22:12:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.export 2023-12-27T22:12:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.export 2023-12-27T22:12:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.export 2023-12-27T22:12:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:12:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.export 2023-12-27T22:12:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_23.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_23.export 2023-12-27T22:12:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:12:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_24 2023-12-27T22:12:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-27T22:12:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:12:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-27T22:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-27T22:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-27T22:12:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:12:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:12:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-27T22:12:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:12:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:12:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:12:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:12:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:12:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:12:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:12:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:12:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-27T22:12:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-27T22:12:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-27T22:12:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:12:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:12:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:12:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:12:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-27T22:12:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-27T22:12:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:12:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:12:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:12:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:12:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:12:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:12:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:12:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:12:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:12:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:12:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:12:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-27T22:12:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:12:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:12:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:12:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:12:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:12:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:12:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:12:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:12:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-27T22:12:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_24.status 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_24.status 2023-12-27T22:12:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.status 2023-12-27T22:12:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_24.status 2023-12-27T22:12:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_24.status 2023-12-27T22:12:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:12:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:12:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-27T22:12:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-27T22:12:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 165 2023-12-27T22:12:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.status 2023-12-27T22:12:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:12:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.status 2023-12-27T22:12:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-27T22:12:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:12:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=165 2023-12-27T22:12:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:12:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:12:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:12:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-27T22:12:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=91 2023-12-27T22:12:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 165 = 91 2023-12-27T22:12:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:12:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:12:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:12:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:12:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:12:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:12:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-27T22:12:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:12:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:12:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:12:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:12:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:12:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:12:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:12:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 165 = '' 2023-12-27T22:12:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:12:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:13:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:13:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:13:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-27T22:13:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 165 = '' 2023-12-27T22:13:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:13:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:13:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:13:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.status 2023-12-27T22:13:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-27T22:13:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:02.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=165 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 165 = 165 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_24': 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109be964abe0 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109be964abe0 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:35 2023 2023-12-27T22:13:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:35 2023 2023-12-27T22:13:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:35 2023 2023-12-27T22:13:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 77a94122-1c65-4804-8b45-6573f18ab770 2023-12-27T22:13:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-27T22:13:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:13:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.export 2023-12-27T22:13:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.export 2023-12-27T22:13:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.export 2023-12-27T22:13:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.export 2023-12-27T22:13:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.export 2023-12-27T22:13:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.export 2023-12-27T22:13:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_24.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_24.export 2023-12-27T22:13:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_25 2023-12-27T22:13:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-27T22:13:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-27T22:13:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:13:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:13:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:13:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:13:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:13:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:13:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-27T22:13:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-27T22:13:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-27T22:13:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:13:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:13:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:13:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-27T22:13:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-27T22:13:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:13:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:13:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:13:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:06.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2023-12-27T22:13:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:13:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:13:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:13:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:13:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:13:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:13:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:13:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-27T22:13:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_25.status 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_25.status 2023-12-27T22:13:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.status 2023-12-27T22:13:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_25.status 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_25.status 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-27T22:13:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 167 2023-12-27T22:13:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.status 2023-12-27T22:13:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.status 2023-12-27T22:13:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-27T22:13:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:13:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=167 2023-12-27T22:13:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:13:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-27T22:13:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=92 2023-12-27T22:13:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 167 = 92 2023-12-27T22:13:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:13:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-27T22:13:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 167 = '' 2023-12-27T22:13:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-27T22:13:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 167 = '' 2023-12-27T22:13:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:13:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.status 2023-12-27T22:13:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-27T22:13:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=167 2023-12-27T22:13:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 167 = 167 2023-12-27T22:13:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-27T22:13:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_25': 2023-12-27T22:13:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109b62b14838 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109b62b14838 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:48 2023 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:48 2023 2023-12-27T22:13:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:48 2023 2023-12-27T22:13:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: ffe3b98c-986e-4f1e-bd91-bea8441372c0 2023-12-27T22:13:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-27T22:13:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:13:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.export 2023-12-27T22:13:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.export 2023-12-27T22:13:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.export 2023-12-27T22:13:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.export 2023-12-27T22:13:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.export 2023-12-27T22:13:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.export 2023-12-27T22:13:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_25.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_25.export 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_26 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:13:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-27T22:13:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:13:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:13:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:13:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:13:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:13:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:13:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:13:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-27T22:13:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-27T22:13:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:13:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:13:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:13:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:13:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-27T22:13:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-27T22:13:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:13:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:13:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:13:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:13:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2023-12-27T22:13:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:13:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:13:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:13:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:13:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-27T22:13:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_26.status 2023-12-27T22:13:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_26.status 2023-12-27T22:13:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.status 2023-12-27T22:13:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_26.status 2023-12-27T22:13:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_26.status 2023-12-27T22:13:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-27T22:13:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-27T22:13:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 169 2023-12-27T22:13:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.status 2023-12-27T22:13:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.status 2023-12-27T22:13:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-27T22:13:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:13:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=169 2023-12-27T22:13:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:13:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-27T22:13:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 169 = '' 2023-12-27T22:13:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:13:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-27T22:13:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 169 = '' 2023-12-27T22:13:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-27T22:13:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 169 = '' 2023-12-27T22:13:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.status 2023-12-27T22:13:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-27T22:13:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=169 2023-12-27T22:13:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 169 = 169 2023-12-27T22:13:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-27T22:13:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_26': 2023-12-27T22:13:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bccda1607 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bccda1607 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:04:57 2023 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:04:57 2023 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:04:57 2023 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 48db3f6c-a1c5-4ad1-b8b2-0b426f65fc7a 2023-12-27T22:13:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.export 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.export 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.export 2023-12-27T22:13:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.export 2023-12-27T22:13:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.export 2023-12-27T22:13:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.export 2023-12-27T22:13:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_26.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_26.export 2023-12-27T22:13:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_27 2023-12-27T22:13:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-27T22:13:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-27T22:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:13:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-27T22:13:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:13:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:13:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-27T22:13:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:13:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:13:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:13:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-27T22:13:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-27T22:13:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:13:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:13:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:13:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-12-27T22:13:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-27T22:13:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:13:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:13:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:13:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-12-27T22:13:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:13:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:13:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:13:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:13:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:13:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_27.status 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_27.status 2023-12-27T22:13:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.status 2023-12-27T22:13:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_27.status 2023-12-27T22:13:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_27.status 2023-12-27T22:13:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-27T22:13:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-27T22:13:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 171 2023-12-27T22:13:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.status 2023-12-27T22:13:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.status 2023-12-27T22:13:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-27T22:13:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:13:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=171 2023-12-27T22:13:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:13:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-27T22:13:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 171 = '' 2023-12-27T22:13:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:13:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-27T22:13:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 171 = '' 2023-12-27T22:13:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-27T22:13:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 171 = '' 2023-12-27T22:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:13:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.status 2023-12-27T22:13:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-27T22:13:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=171 2023-12-27T22:13:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 171 = 171 2023-12-27T22:13:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-27T22:13:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-27T22:13:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_27': 2023-12-27T22:13:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bd572a385 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bd572a385 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:13 2023 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:13 2023 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:13 2023 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 98818c22-b08f-4edd-86c0-d236cb0457ef 2023-12-27T22:13:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.export 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.export 2023-12-27T22:13:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.export 2023-12-27T22:13:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.export 2023-12-27T22:13:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.export 2023-12-27T22:13:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.export 2023-12-27T22:13:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_27.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_27.export 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_28 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:13:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-27T22:13:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-27T22:13:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:13:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-27T22:13:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:13:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:13:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:13:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-27T22:13:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:13:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:13:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:13:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:13:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-27T22:13:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-27T22:13:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-12-27T22:13:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:13:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:13:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:13:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:13:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-12-27T22:13:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-27T22:13:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:13:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:13:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:13:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2023-12-27T22:13:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:13:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:13:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:13:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:13:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:13:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:13:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_28.status 2023-12-27T22:13:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_28.status 2023-12-27T22:13:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.status 2023-12-27T22:13:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_28.status 2023-12-27T22:13:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_28.status 2023-12-27T22:13:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-27T22:13:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-27T22:13:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 173 2023-12-27T22:13:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.status 2023-12-27T22:13:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.status 2023-12-27T22:13:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-27T22:13:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:13:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=173 2023-12-27T22:13:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:13:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-27T22:13:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 173 = '' 2023-12-27T22:13:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:13:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-27T22:13:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 173 = '' 2023-12-27T22:13:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-27T22:13:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 173 = '' 2023-12-27T22:13:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:13:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.status 2023-12-27T22:13:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-27T22:13:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=173 2023-12-27T22:13:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 173 = 173 2023-12-27T22:13:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-27T22:13:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_28': 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109ba2efdcb5 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109ba2efdcb5 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:23 2023 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:23 2023 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:23 2023 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 124b8604-3ea3-41f5-8bc5-de0e5cb86d1f 2023-12-27T22:13:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.export 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.export 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.export 2023-12-27T22:13:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.export 2023-12-27T22:13:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.export 2023-12-27T22:13:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.export 2023-12-27T22:13:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_28.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_28.export 2023-12-27T22:13:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_29 2023-12-27T22:13:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-27T22:13:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:13:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:13:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:13:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:13:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:13:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:13:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:13:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:13:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:13:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:13:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:13:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-27T22:13:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-27T22:13:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:13:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:13:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:13:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:13:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-27T22:13:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-27T22:13:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:13:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:13:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:13:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-12-27T22:13:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:13:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:13:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:13:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:13:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:13:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:13:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:13:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_29.status 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_29.status 2023-12-27T22:13:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.status 2023-12-27T22:13:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_29.status 2023-12-27T22:13:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_29.status 2023-12-27T22:13:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-27T22:13:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-27T22:13:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 175 2023-12-27T22:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.status 2023-12-27T22:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.status 2023-12-27T22:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-27T22:13:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:13:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=175 2023-12-27T22:13:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:13:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-27T22:13:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=96 2023-12-27T22:13:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 175 = 96 2023-12-27T22:13:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:13:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-27T22:13:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 175 = '' 2023-12-27T22:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-27T22:13:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 175 = '' 2023-12-27T22:13:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.status 2023-12-27T22:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-27T22:13:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=175 2023-12-27T22:13:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 175 = 175 2023-12-27T22:13:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_29': 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bab30bb4a 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bab30bb4a 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:35 2023 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:35 2023 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:35 2023 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 916d4730-d7f2-40c8-a499-9224c313b98e 2023-12-27T22:13:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.export 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.export 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.export 2023-12-27T22:13:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.export 2023-12-27T22:13:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.export 2023-12-27T22:13:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.export 2023-12-27T22:13:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_29.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_29.export 2023-12-27T22:13:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_30 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:13:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-27T22:13:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-27T22:13:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:13:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-27T22:13:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:13:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:13:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:13:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:13:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-27T22:13:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:13:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:13:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:13:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:13:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:13:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-27T22:13:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-27T22:13:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:13:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:13:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:13:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-27T22:13:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-27T22:13:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:13:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:13:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:13:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-12-27T22:13:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:13:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:13:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:13:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:13:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-27T22:13:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_30.status 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_30.status 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.status 2023-12-27T22:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_30.status 2023-12-27T22:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_30.status 2023-12-27T22:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-27T22:13:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-27T22:13:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 177 2023-12-27T22:13:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.status 2023-12-27T22:13:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.status 2023-12-27T22:13:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-27T22:13:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:13:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=177 2023-12-27T22:13:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:13:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-27T22:13:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=97 2023-12-27T22:13:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 177 = 97 2023-12-27T22:13:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:13:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-27T22:13:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 177 = '' 2023-12-27T22:13:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-27T22:13:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 177 = '' 2023-12-27T22:13:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:13:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.status 2023-12-27T22:13:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-27T22:13:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=177 2023-12-27T22:13:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 177 = 177 2023-12-27T22:13:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-27T22:13:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_30': 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 109bd28f6e9f 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.109bd28f6e9f 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:05:49 2023 2023-12-27T22:13:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:05:49 2023 2023-12-27T22:13:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:05:49 2023 2023-12-27T22:13:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 505598aa-3a40-4448-9218-b902cec83230 2023-12-27T22:13:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.export 2023-12-27T22:13:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.export 2023-12-27T22:13:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.export 2023-12-27T22:13:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.export 2023-12-27T22:13:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.export 2023-12-27T22:13:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.export 2023-12-27T22:13:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_30.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_30.export 2023-12-27T22:13:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_31 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:13:53.192 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5313414044236969) > weight (0.25) 2023-12-27T22:13:53.192 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5616521705685122) > weight (0.25) 2023-12-27T22:13:53.192 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.37739611598000466) > weight (0.25) 2023-12-27T22:13:53.192 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6670250561946415) > weight (0.25) 2023-12-27T22:13:53.192 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-27T22:13:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:13:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:13:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:13:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:13:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:13:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:13:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:13:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:13:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:13:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:13:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:13:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:13:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:13:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-27T22:13:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-27T22:13:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-12-27T22:13:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:13:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:13:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:13:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:13:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:13:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-12-27T22:13:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-27T22:13:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:13:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:13:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:13:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:13:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:13:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:13:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:13:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:13:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:13:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:13:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:13:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2023-12-27T22:13:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:13:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:13:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:13:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:13:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:13:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:13:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:13:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-27T22:13:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_31.status 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_31.status 2023-12-27T22:13:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.status 2023-12-27T22:13:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_31.status 2023-12-27T22:13:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_31.status 2023-12-27T22:13:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:13:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.status 2023-12-27T22:13:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-27T22:13:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-27T22:13:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 179 2023-12-27T22:13:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.status 2023-12-27T22:13:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.status 2023-12-27T22:13:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-27T22:13:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:13:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=179 2023-12-27T22:13:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:13:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.status 2023-12-27T22:13:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.status 2023-12-27T22:13:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-27T22:13:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=98 2023-12-27T22:13:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 179 = 98 2023-12-27T22:13:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:13:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.status 2023-12-27T22:13:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.status 2023-12-27T22:13:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-27T22:13:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:13:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:13:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:13:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:13:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:13:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 179 = '' 2023-12-27T22:13:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:13:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:13:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.status 2023-12-27T22:13:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.status 2023-12-27T22:13:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-27T22:13:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:13:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=179 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 179 = 179 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:13:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:13:58.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_31': 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288c9ee0b0c 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288c9ee0b0c 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:13:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:01 2023 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:01 2023 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:01 2023 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 103074c2-6a78-4270-a144-22bb49c87496 2023-12-27T22:13:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:13:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.export 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.export 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.export 2023-12-27T22:13:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.export 2023-12-27T22:13:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.export 2023-12-27T22:13:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:13:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.export 2023-12-27T22:13:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_31.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_31.export 2023-12-27T22:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_32 2023-12-27T22:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-27T22:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:13:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:13:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:13:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:13:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-27T22:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:01.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-27T22:14:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-27T22:14:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-12-27T22:14:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-27T22:14:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-12-27T22:14:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-27T22:14:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_32.status 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_32.status 2023-12-27T22:14:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.status 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_32.status 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_32.status 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-27T22:14:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 181 2023-12-27T22:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.status 2023-12-27T22:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.status 2023-12-27T22:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-27T22:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=181 2023-12-27T22:14:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:14:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-27T22:14:03.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=99 2023-12-27T22:14:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 181 = 99 2023-12-27T22:14:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:14:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-27T22:14:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 181 = '' 2023-12-27T22:14:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:03.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:14:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-27T22:14:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 181 = '' 2023-12-27T22:14:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.status 2023-12-27T22:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-27T22:14:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=181 2023-12-27T22:14:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 181 = 181 2023-12-27T22:14:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-27T22:14:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:14:06.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-27T22:14:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_32': 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 128835cbd0f1 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.128835cbd0f1 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:08 2023 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:08 2023 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:08 2023 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:14:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:14:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:14:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 02d41175-0670-4c81-82ed-381c20b18bb9 2023-12-27T22:14:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:14:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-27T22:14:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:14:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:14:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.export 2023-12-27T22:14:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.export 2023-12-27T22:14:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.export 2023-12-27T22:14:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.export 2023-12-27T22:14:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.export 2023-12-27T22:14:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.export 2023-12-27T22:14:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_32.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_32.export 2023-12-27T22:14:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:14:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_33 2023-12-27T22:14:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-27T22:14:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-27T22:14:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-27T22:14:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-27T22:14:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-27T22:14:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-27T22:14:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-27T22:14:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-27T22:14:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-27T22:14:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-27T22:14:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-12-27T22:14:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-12-27T22:14:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-27T22:14:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_33.status 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_33.status 2023-12-27T22:14:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.status 2023-12-27T22:14:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_33.status 2023-12-27T22:14:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_33.status 2023-12-27T22:14:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-27T22:14:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-27T22:14:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 183 2023-12-27T22:14:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.status 2023-12-27T22:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.status 2023-12-27T22:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-27T22:14:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=183 2023-12-27T22:14:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:14:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-27T22:14:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=100 2023-12-27T22:14:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 183 = 100 2023-12-27T22:14:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:14:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-27T22:14:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 183 = '' 2023-12-27T22:14:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:11.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:14:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-27T22:14:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 183 = '' 2023-12-27T22:14:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:14:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.status 2023-12-27T22:14:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-27T22:14:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=183 2023-12-27T22:14:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 183 = 183 2023-12-27T22:14:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-27T22:14:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:14:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-27T22:14:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_33': 2023-12-27T22:14:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:14:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288a402e3c9 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288a402e3c9 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:19 2023 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:19 2023 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:19 2023 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:14:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 232f1d52-1511-44c7-ac99-da55f617e6e4 2023-12-27T22:14:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:14:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.export 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.export 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.export 2023-12-27T22:14:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.export 2023-12-27T22:14:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.export 2023-12-27T22:14:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.export 2023-12-27T22:14:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_33.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_33.export 2023-12-27T22:14:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:14:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_34 2023-12-27T22:14:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-27T22:14:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-27T22:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-27T22:14:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:18.113 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7479460498276898) > weight (0.25) 2023-12-27T22:14:18.113 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.259598283742165) > weight (0.25) 2023-12-27T22:14:18.113 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5872476730756262) > weight (0.25) 2023-12-27T22:14:18.114 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6304452580822301) > weight (0.25) 2023-12-27T22:14:18.114 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-27T22:14:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-27T22:14:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-27T22:14:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-27T22:14:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-27T22:14:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-27T22:14:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-27T22:14:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-27T22:14:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_34.status 2023-12-27T22:14:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_34.status 2023-12-27T22:14:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.status 2023-12-27T22:14:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_34.status 2023-12-27T22:14:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_34.status 2023-12-27T22:14:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-27T22:14:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-27T22:14:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 185 2023-12-27T22:14:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.status 2023-12-27T22:14:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.status 2023-12-27T22:14:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-27T22:14:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=185 2023-12-27T22:14:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:14:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-27T22:14:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=101 2023-12-27T22:14:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 185 = 101 2023-12-27T22:14:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:14:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-27T22:14:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 185 = '' 2023-12-27T22:14:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:14:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-27T22:14:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 185 = '' 2023-12-27T22:14:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.status 2023-12-27T22:14:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-27T22:14:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=185 2023-12-27T22:14:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 185 = 185 2023-12-27T22:14:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-27T22:14:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:23.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:14:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-27T22:14:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_34': 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12889c7c3e1e 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12889c7c3e1e 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:27 2023 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:27 2023 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:27 2023 2023-12-27T22:14:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:14:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:14:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:14:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 619c2405-7b4e-4332-9bd5-b1bd4115bc7d 2023-12-27T22:14:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:14:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-27T22:14:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:14:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:14:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.export 2023-12-27T22:14:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.export 2023-12-27T22:14:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.export 2023-12-27T22:14:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.export 2023-12-27T22:14:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.export 2023-12-27T22:14:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.export 2023-12-27T22:14:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_34.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_34.export 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_35 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-27T22:14:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-12-27T22:14:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-27T22:14:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-27T22:14:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-12-27T22:14:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-27T22:14:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2023-12-27T22:14:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-27T22:14:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_35.status 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_35.status 2023-12-27T22:14:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.status 2023-12-27T22:14:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_35.status 2023-12-27T22:14:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_35.status 2023-12-27T22:14:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-27T22:14:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-27T22:14:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 187 2023-12-27T22:14:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.status 2023-12-27T22:14:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.status 2023-12-27T22:14:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-27T22:14:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=187 2023-12-27T22:14:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:27.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:14:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-27T22:14:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=102 2023-12-27T22:14:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 187 = 102 2023-12-27T22:14:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:14:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-27T22:14:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 187 = '' 2023-12-27T22:14:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:14:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-27T22:14:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 187 = '' 2023-12-27T22:14:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:14:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.status 2023-12-27T22:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-27T22:14:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=187 2023-12-27T22:14:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 187 = 187 2023-12-27T22:14:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-27T22:14:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:14:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_35': 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12886fdf8cff 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12886fdf8cff 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:35 2023 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:35 2023 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:35 2023 2023-12-27T22:14:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:14:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:14:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:14:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: abdbf0b7-80a9-450d-bd6c-9b78f7665b62 2023-12-27T22:14:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:14:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-27T22:14:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:14:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:14:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.export 2023-12-27T22:14:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.export 2023-12-27T22:14:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.export 2023-12-27T22:14:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.export 2023-12-27T22:14:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.export 2023-12-27T22:14:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.export 2023-12-27T22:14:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_35.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_35.export 2023-12-27T22:14:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:14:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_36 2023-12-27T22:14:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-27T22:14:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-27T22:14:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-27T22:14:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-27T22:14:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-27T22:14:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-12-27T22:14:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-12-27T22:14:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-27T22:14:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2023-12-27T22:14:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-27T22:14:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_36.status 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_36.status 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.status 2023-12-27T22:14:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_36.status 2023-12-27T22:14:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_36.status 2023-12-27T22:14:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-27T22:14:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-27T22:14:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 189 2023-12-27T22:14:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.status 2023-12-27T22:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.status 2023-12-27T22:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-27T22:14:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=189 2023-12-27T22:14:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:35.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:14:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-27T22:14:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=103 2023-12-27T22:14:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 189 = 103 2023-12-27T22:14:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:14:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-27T22:14:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 189 = '' 2023-12-27T22:14:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:14:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-27T22:14:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 189 = '' 2023-12-27T22:14:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:14:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.status 2023-12-27T22:14:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-27T22:14:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=189 2023-12-27T22:14:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 189 = 189 2023-12-27T22:14:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-27T22:14:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:14:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-27T22:14:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_36': 2023-12-27T22:14:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:14:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288f83b468b 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288f83b468b 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:42 2023 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:42 2023 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:42 2023 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:14:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:14:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: d7e1cda4-b8f6-4e1c-9d85-6eaadd6f32c4 2023-12-27T22:14:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.export 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.export 2023-12-27T22:14:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.export 2023-12-27T22:14:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.export 2023-12-27T22:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.export 2023-12-27T22:14:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.export 2023-12-27T22:14:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_36.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_36.export 2023-12-27T22:14:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:14:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_37 2023-12-27T22:14:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-27T22:14:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-27T22:14:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-27T22:14:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-12-27T22:14:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-27T22:14:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:43.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2023-12-27T22:14:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_37.status 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_37.status 2023-12-27T22:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.status 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_37.status 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_37.status 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-27T22:14:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 191 2023-12-27T22:14:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.status 2023-12-27T22:14:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.status 2023-12-27T22:14:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-27T22:14:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=191 2023-12-27T22:14:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:14:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-27T22:14:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=104 2023-12-27T22:14:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 191 = 104 2023-12-27T22:14:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:14:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-27T22:14:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 191 = '' 2023-12-27T22:14:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:14:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-27T22:14:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 191 = '' 2023-12-27T22:14:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:14:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.status 2023-12-27T22:14:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-27T22:14:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=191 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 191 = 191 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:14:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-27T22:14:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_37': 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288b97af3ca 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288b97af3ca 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:53 2023 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:53 2023 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:53 2023 2023-12-27T22:14:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:14:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:14:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:14:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: cb7251f0-99f4-4c03-a067-b3a87657c534 2023-12-27T22:14:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:14:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-27T22:14:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:14:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:14:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.export 2023-12-27T22:14:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.export 2023-12-27T22:14:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.export 2023-12-27T22:14:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.export 2023-12-27T22:14:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.export 2023-12-27T22:14:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.export 2023-12-27T22:14:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_37.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_37.export 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_38 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-27T22:14:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-27T22:14:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-27T22:14:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-27T22:14:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-12-27T22:14:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-12-27T22:14:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-12-27T22:14:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-12-27T22:14:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_38.status 2023-12-27T22:14:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_38.status 2023-12-27T22:14:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.status 2023-12-27T22:14:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_38.status 2023-12-27T22:14:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_38.status 2023-12-27T22:14:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-27T22:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-27T22:14:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 193 2023-12-27T22:14:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.status 2023-12-27T22:14:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.status 2023-12-27T22:14:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-27T22:14:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=193 2023-12-27T22:14:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:14:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-27T22:14:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=105 2023-12-27T22:14:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 193 = 105 2023-12-27T22:14:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:14:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-27T22:14:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 193 = '' 2023-12-27T22:14:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:14:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-27T22:14:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:14:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:14:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:14:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:14:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 193 = '' 2023-12-27T22:14:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:14:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.status 2023-12-27T22:14:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-27T22:14:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:14:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=193 2023-12-27T22:14:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 193 = 193 2023-12-27T22:14:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-27T22:14:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:14:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-27T22:14:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_38': 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12888a152104 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12888a152104 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:06:59 2023 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:06:59 2023 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:06:59 2023 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:14:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:14:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: a570017e-0f25-4b1e-bf55-91e5287082df 2023-12-27T22:14:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:14:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-27T22:14:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:14:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:14:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.export 2023-12-27T22:14:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.export 2023-12-27T22:14:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.export 2023-12-27T22:14:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.export 2023-12-27T22:14:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.export 2023-12-27T22:14:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:14:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.export 2023-12-27T22:14:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_38.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_38.export 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_39 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:14:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:14:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:14:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:14:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:14:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:14:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:14:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:14:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:14:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:14:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:14:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:14:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:14:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:14:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-27T22:14:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-27T22:14:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-27T22:14:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:14:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:14:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:14:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:14:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:14:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-27T22:14:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-27T22:14:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:14:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:14:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:14:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:14:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:14:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:14:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2023-12-27T22:14:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:14:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:14:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:14:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:14:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:14:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_39.status 2023-12-27T22:14:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_39.status 2023-12-27T22:14:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.status 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_39.status 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_39.status 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-27T22:14:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 195 2023-12-27T22:14:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.status 2023-12-27T22:14:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:14:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:14:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:14:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.status 2023-12-27T22:14:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-12-27T22:14:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:14:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:14:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=195 2023-12-27T22:14:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:14:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:14:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:15:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-27T22:15:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:00.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=106 2023-12-27T22:15:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 195 = 106 2023-12-27T22:15:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:15:00.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-27T22:15:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 195 = '' 2023-12-27T22:15:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:15:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-27T22:15:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 195 = '' 2023-12-27T22:15:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:15:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:15:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.status 2023-12-27T22:15:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-27T22:15:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=195 2023-12-27T22:15:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 195 = 195 2023-12-27T22:15:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-12-27T22:15:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:15:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:15:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-12-27T22:15:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_39': 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288cb922074 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288cb922074 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:11 2023 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:11 2023 2023-12-27T22:15:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:11 2023 2023-12-27T22:15:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:15:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:15:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:15:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 4cb400ed-6f6e-497a-ba79-d1988e890173 2023-12-27T22:15:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:15:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_39 snap 2023-12-27T22:15:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:15:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:15:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.export 2023-12-27T22:15:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.export 2023-12-27T22:15:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.export 2023-12-27T22:15:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.export 2023-12-27T22:15:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.export 2023-12-27T22:15:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.export 2023-12-27T22:15:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_39.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_39.export 2023-12-27T22:15:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:15:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_40 2023-12-27T22:15:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_40 snap 2023-12-27T22:15:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:15:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-12-27T22:15:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-12-27T22:15:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:15:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-12-27T22:15:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-12-27T22:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:15:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:15:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:15:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-12-27T22:15:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-12-27T22:15:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-27T22:15:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-12-27T22:15:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:15:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:15:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:15:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:15:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-12-27T22:15:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-27T22:15:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-12-27T22:15:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:15:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:15:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:15:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:15:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:15:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-12-27T22:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:15:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:15:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:15:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:15:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:15:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:15:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-12-27T22:15:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:15:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:15:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-12-27T22:15:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_40.status 2023-12-27T22:15:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_40.status 2023-12-27T22:15:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.status 2023-12-27T22:15:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_40.status 2023-12-27T22:15:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_40.status 2023-12-27T22:15:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-12-27T22:15:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-12-27T22:15:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 197 2023-12-27T22:15:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.status 2023-12-27T22:15:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.status 2023-12-27T22:15:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-12-27T22:15:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=197 2023-12-27T22:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-27T22:15:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=107 2023-12-27T22:15:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 197 = 107 2023-12-27T22:15:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-27T22:15:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 197 = '' 2023-12-27T22:15:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-27T22:15:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 197 = '' 2023-12-27T22:15:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.status 2023-12-27T22:15:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-27T22:15:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=197 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 197 = 197 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:15:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-12-27T22:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_40': 2023-12-27T22:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:15:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288ed81e651 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288ed81e651 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:21 2023 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:21 2023 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:21 2023 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 0b9a6172-e596-4e63-ada8-38dbbc7d188d 2023-12-27T22:15:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:15:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_40 snap 2023-12-27T22:15:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:15:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.export 2023-12-27T22:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.export 2023-12-27T22:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.export 2023-12-27T22:15:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.export 2023-12-27T22:15:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.export 2023-12-27T22:15:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.export 2023-12-27T22:15:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_40.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_40.export 2023-12-27T22:15:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:15:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_41 2023-12-27T22:15:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_41 snap 2023-12-27T22:15:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:15:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-12-27T22:15:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:15:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:15:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:15:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-12-27T22:15:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:15:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:15:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:15:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:15:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:15:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:15:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-12-27T22:15:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-27T22:15:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:15:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:15:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:15:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-12-27T22:15:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-27T22:15:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:15:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:15:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:15:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2023-12-27T22:15:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:15:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:15:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:15:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:15:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_41.status 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_41.status 2023-12-27T22:15:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.status 2023-12-27T22:15:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_41.status 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_41.status 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-12-27T22:15:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 199 2023-12-27T22:15:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.status 2023-12-27T22:15:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.status 2023-12-27T22:15:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-12-27T22:15:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:15:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=199 2023-12-27T22:15:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-27T22:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=108 2023-12-27T22:15:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 199 = 108 2023-12-27T22:15:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-27T22:15:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 199 = '' 2023-12-27T22:15:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-27T22:15:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 199 = '' 2023-12-27T22:15:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:15:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.status 2023-12-27T22:15:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-27T22:15:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=199 2023-12-27T22:15:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 199 = 199 2023-12-27T22:15:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-12-27T22:15:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:15:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_41': 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288ca24e31e 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288ca24e31e 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:15:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:28 2023 2023-12-27T22:15:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:28 2023 2023-12-27T22:15:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:28 2023 2023-12-27T22:15:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:15:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:15:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:15:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 8f628a70-f2b5-4041-a2d4-bbbc883caac7 2023-12-27T22:15:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:15:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_41 snap 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.export 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.export 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.export 2023-12-27T22:15:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.export 2023-12-27T22:15:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.export 2023-12-27T22:15:21.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.export 2023-12-27T22:15:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_41.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_41.export 2023-12-27T22:15:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_42 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_42 snap 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:15:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-12-27T22:15:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-12-27T22:15:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:15:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-12-27T22:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:15:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-12-27T22:15:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-27T22:15:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:15:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2023-12-27T22:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-27T22:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:15:24.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:15:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:15:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:15:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2023-12-27T22:15:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:15:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:15:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:15:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:15:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_42.status 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_42.status 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.status 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_42.status 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_42.status 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-12-27T22:15:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 201 2023-12-27T22:15:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.status 2023-12-27T22:15:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.status 2023-12-27T22:15:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-12-27T22:15:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:15:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=201 2023-12-27T22:15:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-27T22:15:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:25.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 201 = '' 2023-12-27T22:15:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-27T22:15:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 201 = '' 2023-12-27T22:15:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-27T22:15:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 201 = '' 2023-12-27T22:15:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:15:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.status 2023-12-27T22:15:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-27T22:15:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=201 2023-12-27T22:15:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 201 = 201 2023-12-27T22:15:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-12-27T22:15:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:15:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:29.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8359878509467801) > weight (0.25) 2023-12-27T22:15:29.194 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5303492506911917) > weight (0.25) 2023-12-27T22:15:29.194 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.45244308311629633) > weight (0.25) 2023-12-27T22:15:29.194 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4546990949074642) > weight (0.25) 2023-12-27T22:15:29.194 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-12-27T22:15:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-12-27T22:15:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_42': 2023-12-27T22:15:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288a00d9725 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288a00d9725 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:38 2023 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:38 2023 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:38 2023 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:15:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:15:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 7b5df11a-e491-4959-be66-344df8a2e44a 2023-12-27T22:15:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_42 snap 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.export 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.export 2023-12-27T22:15:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.export 2023-12-27T22:15:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.export 2023-12-27T22:15:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.export 2023-12-27T22:15:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.export 2023-12-27T22:15:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_42.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_42.export 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_43 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_43 snap 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:15:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-12-27T22:15:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-12-27T22:15:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:15:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:15:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:15:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-12-27T22:15:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-12-27T22:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-27T22:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-12-27T22:15:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:15:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:15:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:15:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-12-27T22:15:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-27T22:15:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2023-12-27T22:15:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:15:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:15:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:15:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:15:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:15:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2023-12-27T22:15:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:15:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:15:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:15:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:15:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_43.status 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_43.status 2023-12-27T22:15:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.status 2023-12-27T22:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_43.status 2023-12-27T22:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_43.status 2023-12-27T22:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-12-27T22:15:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-12-27T22:15:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 203 2023-12-27T22:15:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.status 2023-12-27T22:15:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.status 2023-12-27T22:15:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-12-27T22:15:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:15:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=203 2023-12-27T22:15:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-27T22:15:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 203 = '' 2023-12-27T22:15:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-27T22:15:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:33.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:33.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 203 = '' 2023-12-27T22:15:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-27T22:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 203 = '' 2023-12-27T22:15:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:15:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.status 2023-12-27T22:15:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-27T22:15:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=203 2023-12-27T22:15:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 203 = 203 2023-12-27T22:15:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-12-27T22:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:15:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-12-27T22:15:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_43': 2023-12-27T22:15:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:15:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 128872c878c2 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.128872c878c2 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:07:52 2023 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:07:52 2023 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:07:52 2023 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 6303d43d-7367-464b-9985-add1a3b57264 2023-12-27T22:15:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_43 snap 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.export 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.export 2023-12-27T22:15:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.export 2023-12-27T22:15:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.export 2023-12-27T22:15:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.export 2023-12-27T22:15:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.export 2023-12-27T22:15:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_43.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_43.export 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_44 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_44 snap 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-12-27T22:15:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:15:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:15:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:15:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-12-27T22:15:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:15:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:15:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:15:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:40.024 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-27T22:15:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-27T22:15:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:15:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:15:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:15:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-12-27T22:15:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-12-27T22:15:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-12-27T22:15:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:15:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:15:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:15:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:15:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-12-27T22:15:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:15:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:15:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:15:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:15:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:15:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_44.status 2023-12-27T22:15:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_44.status 2023-12-27T22:15:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.status 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_44.status 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_44.status 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.status 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-12-27T22:15:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 205 2023-12-27T22:15:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.status 2023-12-27T22:15:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.status 2023-12-27T22:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-12-27T22:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=205 2023-12-27T22:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.status 2023-12-27T22:15:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.status 2023-12-27T22:15:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-27T22:15:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 205 = '' 2023-12-27T22:15:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.status 2023-12-27T22:15:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.status 2023-12-27T22:15:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-27T22:15:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 205 = '' 2023-12-27T22:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.status 2023-12-27T22:15:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.status 2023-12-27T22:15:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-27T22:15:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=205 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 205 = 205 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:15:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:43.115 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.682572833098156) > weight (0.25) 2023-12-27T22:15:43.115 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7985650521883795) > weight (0.25) 2023-12-27T22:15:43.115 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-27T22:15:43.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-27T22:15:43.116 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-27T22:15:43.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:2023-12-27T22:15:43.117+0000 7fa2629e6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 61100) UID: 1000 2023-12-27T22:15:43.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:15:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_44': 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12886d100951 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12886d100951 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:15:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:02 2023 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:02 2023 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:02 2023 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 5ce59573-6607-4c36-b8ca-ffaa5d72c073 2023-12-27T22:15:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_44 snap 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.export 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.export 2023-12-27T22:15:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.export 2023-12-27T22:15:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.export 2023-12-27T22:15:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.export 2023-12-27T22:15:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.export 2023-12-27T22:15:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_44.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_44.export 2023-12-27T22:15:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_45 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_45 snap 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:15:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:15:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:15:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:15:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-27T22:15:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-27T22:15:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:1 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:1 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:1 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=1 2023-12-27T22:15:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-27T22:15:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=1 2023-12-27T22:15:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=1 2023-12-27T22:15:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-27T22:15:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-27T22:15:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-12-27T22:15:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-27T22:15:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2023-12-27T22:15:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:2 2023-12-27T22:15:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-27T22:15:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:2 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=2 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=2 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:2 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:2 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=2 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=2 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=2 2023-12-27T22:15:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-27T22:15:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-27T22:15:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:3 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-27T22:15:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:3 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=3 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=3 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-27T22:15:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:3 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:3 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=3 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=3 2023-12-27T22:15:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=3 2023-12-27T22:15:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-27T22:15:47.444 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:15:47.444+0000 7f67e642e700 -1 reset not still connected to 0x55ca79b44410 2023-12-27T22:15:47.444 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:15:47.444+0000 7f67e642e700 -1 reset not still connected to 0x55ca79b44b60 2023-12-27T22:15:47.445 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:15:47.445+0000 7f67e642e700 -1 reset not still connected to 0x55ca79b44d00 2023-12-27T22:15:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:15:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:15:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-27T22:15:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2023-12-27T22:15:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-27T22:15:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd mirror status : get status for rbd mirror 2023-12-27T22:15:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:admin_socket: invalid command 2023-12-27T22:15:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 1 2023-12-27T22:15:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-27T22:15:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:15:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:15:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:15:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:15:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_45.status 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_45.status 2023-12-27T22:15:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.status 2023-12-27T22:15:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_45.status 2023-12-27T22:15:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_45.status 2023-12-27T22:15:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-12-27T22:15:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-12-27T22:15:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 207 2023-12-27T22:15:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.status 2023-12-27T22:15:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.status 2023-12-27T22:15:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-12-27T22:15:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:15:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=207 2023-12-27T22:15:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-27T22:15:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 207 = '' 2023-12-27T22:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-27T22:15:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 207 = '' 2023-12-27T22:15:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:50.117 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-27T22:15:50.117 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:15:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:15:50.118 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-27T22:15:50.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-27T22:15:50.118 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-27T22:15:50.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-27T22:15:50.121 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-27T22:15:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-27T22:15:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 207 = '' 2023-12-27T22:15:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:15:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.status 2023-12-27T22:15:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-27T22:15:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=207 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 207 = 207 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:15:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-12-27T22:15:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_45': 2023-12-27T22:15:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:15:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:15:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 128888a8e077 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.128888a8e077 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:12 2023 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:12 2023 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:12 2023 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:15:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: c1569f06-8c36-41d6-83d4-36589c3c6c54 2023-12-27T22:15:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_45 snap 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.export 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.export 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.export 2023-12-27T22:15:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.export 2023-12-27T22:15:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.export 2023-12-27T22:15:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:15:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.export 2023-12-27T22:15:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_45.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_45.export 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_46 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_46 snap 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:15:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:15:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-12-27T22:15:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-12-27T22:15:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:15:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-12-27T22:15:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:15:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:15:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:15:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:15:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:15:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:15:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:15:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:15:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:15:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-12-27T22:15:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:15:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:15:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:15:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:15:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-12-27T22:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_46.status 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_46.status 2023-12-27T22:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.status 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_46.status 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_46.status 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-12-27T22:15:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 209 2023-12-27T22:15:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.status 2023-12-27T22:15:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:15:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.status 2023-12-27T22:15:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-12-27T22:15:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:15:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=209 2023-12-27T22:15:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:15:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:15:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:15:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-27T22:15:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 209 = '' 2023-12-27T22:15:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:15:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:15:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:15:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-27T22:15:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 209 = '' 2023-12-27T22:15:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-27T22:15:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:15:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:15:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 209 = '' 2023-12-27T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:15:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1.6 2023-12-27T22:16:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:16:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:16:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.status 2023-12-27T22:16:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-27T22:16:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=209 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 209 = 209 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:16:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_46': 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288c364c368 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288c364c368 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:16:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:19 2023 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:19 2023 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:19 2023 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 79fc8fc4-a92b-41e3-8510-d70489a1905a 2023-12-27T22:16:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:16:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_46 snap 2023-12-27T22:16:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:16:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:16:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.export 2023-12-27T22:16:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.export 2023-12-27T22:16:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.export 2023-12-27T22:16:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.export 2023-12-27T22:16:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.export 2023-12-27T22:16:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.export 2023-12-27T22:16:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_46.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_46.export 2023-12-27T22:16:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_47 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_47 snap 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:16:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-12-27T22:16:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-12-27T22:16:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:16:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-12-27T22:16:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:16:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:16:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:16:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-12-27T22:16:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:16:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:16:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:16:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:16:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:16:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-27T22:16:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:16:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:16:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:16:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:16:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:16:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:16:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:16:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-27T22:16:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:16:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:16:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:16:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:16:04.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:16:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:16:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:16:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-12-27T22:16:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_47.status 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_47.status 2023-12-27T22:16:04.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.status 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_47.status 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_47.status 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.status 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-12-27T22:16:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 211 2023-12-27T22:16:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.status 2023-12-27T22:16:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.status 2023-12-27T22:16:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-12-27T22:16:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:16:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=211 2023-12-27T22:16:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:16:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.status 2023-12-27T22:16:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.status 2023-12-27T22:16:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-27T22:16:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 211 = '' 2023-12-27T22:16:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:16:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.status 2023-12-27T22:16:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.status 2023-12-27T22:16:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-27T22:16:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 211 = '' 2023-12-27T22:16:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:16:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.status 2023-12-27T22:16:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.status 2023-12-27T22:16:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-27T22:16:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=211 2023-12-27T22:16:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 211 = 211 2023-12-27T22:16:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-12-27T22:16:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:16:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-12-27T22:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_47': 2023-12-27T22:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288ed3c2969 2023-12-27T22:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288ed3c2969 2023-12-27T22:16:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:26 2023 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:26 2023 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:26 2023 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: b49aa2f8-63aa-40d1-91ed-40c882361a94 2023-12-27T22:16:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:16:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_47 snap 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.export 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.export 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.export 2023-12-27T22:16:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.export 2023-12-27T22:16:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.export 2023-12-27T22:16:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.export 2023-12-27T22:16:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_47.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_47.export 2023-12-27T22:16:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_48 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_48 snap 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:16:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-12-27T22:16:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:16:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:16:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:16:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:16:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:16:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-12-27T22:16:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:16:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:16:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:16:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:16:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:16:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:16:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:16:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:16:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:16:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:16:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:16:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:16:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-12-27T22:16:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:16:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:16:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_48.status 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_48.status 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.status 2023-12-27T22:16:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_48.status 2023-12-27T22:16:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_48.status 2023-12-27T22:16:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.status 2023-12-27T22:16:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-12-27T22:16:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-12-27T22:16:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 213 2023-12-27T22:16:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.status 2023-12-27T22:16:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.status 2023-12-27T22:16:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-12-27T22:16:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=213 2023-12-27T22:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.status 2023-12-27T22:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.status 2023-12-27T22:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-27T22:16:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 213 = '' 2023-12-27T22:16:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:16:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.status 2023-12-27T22:16:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.status 2023-12-27T22:16:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-27T22:16:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 213 = '' 2023-12-27T22:16:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:16:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.status 2023-12-27T22:16:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.status 2023-12-27T22:16:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-27T22:16:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=213 2023-12-27T22:16:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 213 = 213 2023-12-27T22:16:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:16:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-12-27T22:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_48': 2023-12-27T22:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1288a6a60618 2023-12-27T22:16:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1288a6a60618 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:35 2023 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:35 2023 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:35 2023 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 526175a8-e0bc-47f4-b966-8b42df9ad7e6 2023-12-27T22:16:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_48 snap 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.export 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.export 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.export 2023-12-27T22:16:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.export 2023-12-27T22:16:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.export 2023-12-27T22:16:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.export 2023-12-27T22:16:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_48.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_48.export 2023-12-27T22:16:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_49 2023-12-27T22:16:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_49 snap 2023-12-27T22:16:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:16:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-12-27T22:16:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-12-27T22:16:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:16:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-12-27T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:16:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:16:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:16:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:16:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:16:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:16:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:16:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:16:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:16:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:16:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:16:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-12-27T22:16:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:16:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:16:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:16:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:16:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:16:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:16:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:16:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_49.status 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_49.status 2023-12-27T22:16:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.status 2023-12-27T22:16:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_49.status 2023-12-27T22:16:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_49.status 2023-12-27T22:16:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.status 2023-12-27T22:16:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-12-27T22:16:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-12-27T22:16:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 215 2023-12-27T22:16:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.status 2023-12-27T22:16:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.status 2023-12-27T22:16:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-12-27T22:16:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:16:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=215 2023-12-27T22:16:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:16:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.status 2023-12-27T22:16:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.status 2023-12-27T22:16:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-27T22:16:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 215 = '' 2023-12-27T22:16:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:16:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.status 2023-12-27T22:16:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.status 2023-12-27T22:16:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-27T22:16:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 215 = '' 2023-12-27T22:16:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:16:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.status 2023-12-27T22:16:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.status 2023-12-27T22:16:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-27T22:16:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=215 2023-12-27T22:16:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 215 = 215 2023-12-27T22:16:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:16:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-12-27T22:16:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_49': 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12883452a3b0 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12883452a3b0 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:51 2023 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:51 2023 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:51 2023 2023-12-27T22:16:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:16:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:16:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:16:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 13149630-0d06-4f7b-80ad-5c081ac5165b 2023-12-27T22:16:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_49 snap 2023-12-27T22:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.export 2023-12-27T22:16:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.export 2023-12-27T22:16:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.export 2023-12-27T22:16:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.export 2023-12-27T22:16:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.export 2023-12-27T22:16:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.export 2023-12-27T22:16:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_49.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_49.export 2023-12-27T22:16:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_50 2023-12-27T22:16:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_snap cluster2 mirror image_50 snap 2023-12-27T22:16:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-12-27T22:16:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=started 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=started 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_result 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=stopped 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_instance=cluster1 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance=cluster1 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local loop_instance 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep -i state 2023-12-27T22:16:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-27T22:16:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sleep 0 2023-12-27T22:16:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-27T22:16:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ seq 0 3 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1:0 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ shift 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local val=cluster1:0 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster_var_name=cluster 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local instance_var_name=instance 2023-12-27T22:16:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval cluster=cluster1 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ eval instance=0 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local local_cluster=cluster1:0 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster=cluster1 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local val=cluster1:0 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local cluster_var_name=local_cluster 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ local instance_var_name=instance 2023-12-27T22:16:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ cluster=cluster1 2023-12-27T22:16:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ instance=0 2023-12-27T22:16:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-27T22:16:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval local_cluster=cluster1 2023-12-27T22:16:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ local_cluster=cluster1 2023-12-27T22:16:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ eval instance=0 2023-12-27T22:16:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ instance=0 2023-12-27T22:16:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-27T22:16:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-27T22:16:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-12-27T22:16:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ return 0 2023-12-27T22:16:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ status_result=' "state": "Replaying"' 2023-12-27T22:16:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo ' "state": "Replaying"' 2023-12-27T22:16:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ grep -i Replaying 2023-12-27T22:16:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: "state": "Replaying" 2023-12-27T22:16:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=started 2023-12-27T22:16:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test started = started 2023-12-27T22:16:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = journal ']' 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' snapshot = snapshot ']' 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_cluster=cluster1 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster2-mirror-image_50.status 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster2-mirror-image_50.status 2023-12-27T22:16:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local status_log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.status 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ mkfname cluster1-mirror-image_50.status 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo cluster1-mirror-image_50.status 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local local_status_log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.status 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-12-27T22:16:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Snapshot ID: 217 2023-12-27T22:16:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.status 2023-12-27T22:16:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.status 2023-12-27T22:16:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-12-27T22:16:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-27T22:16:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snapshot_id=217 2023-12-27T22:16:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.2 2023-12-27T22:16:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.status 2023-12-27T22:16:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.status 2023-12-27T22:16:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-27T22:16:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 217 = '' 2023-12-27T22:16:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.4 2023-12-27T22:16:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.status 2023-12-27T22:16:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.status 2023-12-27T22:16:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-27T22:16:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:16:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:-:1.1: Document is empty 2023-12-27T22:16:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: 2023-12-27T22:16:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:^ 2023-12-27T22:16:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id= 2023-12-27T22:16:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 217 = '' 2023-12-27T22:16:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-27T22:16:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.8 2023-12-27T22:16:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.status 2023-12-27T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local log=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.status 2023-12-27T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-27T22:16:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-27T22:16:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-27T22:16:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local primary_snapshot_id=217 2023-12-27T22:16:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 217 = 217 2023-12-27T22:16:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-12-27T22:16:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:16:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:16:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-27T22:16:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 1 2023-12-27T22:16:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_50': 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 128 MiB in 32 objects 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 2 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 12888f6ba228 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.12888f6ba228 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:08:59 2023 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:08:59 2023 2023-12-27T22:16:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:08:59 2023 2023-12-27T22:16:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: protected: False 2023-12-27T22:16:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:16:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:16:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 8f0ce147-0f9e-4455-a07c-cb25053ca356 2023-12-27T22:16:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: false 2023-12-27T22:16:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ compare_image_snaps mirror image_50 snap 2023-12-27T22:16:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:16:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap_name=snap 2023-12-27T22:16:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local ret=0 2023-12-27T22:16:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local rmt_export=/tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.export 2023-12-27T22:16:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local loc_export=/tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.export 2023-12-27T22:16:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.export 2023-12-27T22:16:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.export 2023-12-27T22:16:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.export 2023-12-27T22:16:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-27T22:16:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cmp /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.export 2023-12-27T22:16:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rm -f /tmp/tmp.yuhsqMaECB/cluster2-mirror-image_50.export /tmp/tmp.yuhsqMaECB/cluster1-mirror-image_50.export 2023-12-27T22:16:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ testlog 'TEST: delete many images' 2023-12-27T22:16:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee -a /tmp/tmp.yuhsqMaECB/rbd-mirror.test.log 2023-12-27T22:16:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ date '+%F %T' 2023-12-27T22:16:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo 2023-12-27 22:16:29 TEST: delete many images 2023-12-27T22:16:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-27 22:16:29 TEST: delete many images 2023-12-27T22:16:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 50 2023-12-27T22:16:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_1 2023-12-27T22:16:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-12-27T22:16:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:16:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-12-27T22:16:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-27T22:16:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:16:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_1 2023-12-27T22:16:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:16:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-27T22:16:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-27T22:16:30.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:30.325+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997f35380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:30.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:30.325+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bdac80 [3/42426da1-e209-4023-8d5f-43ff971b1cd6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:30.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:30.384+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562995ef2820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:30.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:30.384+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997bdac80 [3/42426da1-e209-4023-8d5f-43ff971b1cd6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:32.297 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_2 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-12-27T22:16:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-27T22:16:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:16:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_2 2023-12-27T22:16:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:16:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-27T22:16:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-27T22:16:33.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:33.342+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997f35ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:33.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:33.342+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997db4a00 [3/f4c094d4-f920-4b5f-ade9-72239f34d084] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:34.249 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:16:34.248+0000 7fa0daabc700 -1 reset not still connected to 0x5567137cbe10 2023-12-27T22:16:34.249 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:16:34.248+0000 7fa0daabc700 -1 reset not still connected to 0x55671386eb60 2023-12-27T22:16:34.249 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:16:34.248+0000 7fa0daabc700 -1 reset not still connected to 0x55671386eea0 2023-12-27T22:16:34.249 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:16:34.248+0000 7fa0daabc700 -1 reset not still connected to 0x55671386f040 2023-12-27T22:16:34.250 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:16:34.249+0000 7fa0daabc700 -1 reset not still connected to 0x556713e61c70 2023-12-27T22:16:34.250 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:16:34.249+0000 7fa0daabc700 -1 reset not still connected to 0x556714353d40 2023-12-27T22:16:35.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_3 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-12-27T22:16:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-27T22:16:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:16:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_3 2023-12-27T22:16:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:16:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-27T22:16:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-27T22:16:36.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:36.359+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d32d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:36.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:36.359+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997d4b900 [3/d21ceb08-f137-46c9-a16b-ae39e2b1220d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_4 2023-12-27T22:16:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-12-27T22:16:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:16:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-12-27T22:16:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-27T22:16:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:16:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_4 2023-12-27T22:16:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:16:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-27T22:16:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-27T22:16:39.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:39.375+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d32ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:39.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:39.375+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997d4af00 [3/c0d17036-eff0-4639-8b58-29f45ac3edc8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:40.195 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5944072198188566) > weight (0.25) 2023-12-27T22:16:40.195 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9780383141538688) > weight (0.25) 2023-12-27T22:16:40.195 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9202196200450251) > weight (0.25) 2023-12-27T22:16:40.195 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.26555194930876846) > weight (0.25) 2023-12-27T22:16:40.195 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2023-12-27T22:16:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_5 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:16:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-12-27T22:16:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-27T22:16:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:16:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_5 2023-12-27T22:16:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:16:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-27T22:16:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-27T22:16:41.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:41.985+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562994878680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:41.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:41.985+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997d4bb80 [3/d2ad7245-32f6-4c55-975e-2cc6979edf3e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_6 2023-12-27T22:16:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-12-27T22:16:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:16:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-12-27T22:16:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-27T22:16:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:16:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_6 2023-12-27T22:16:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:16:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-27T22:16:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-27T22:16:45.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:45.001+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b8a1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:45.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:45.001+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997c07180 [3/7b6aaf8f-abec-4e77-a74c-774b6db907b0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_7 2023-12-27T22:16:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-12-27T22:16:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:16:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-12-27T22:16:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-27T22:16:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:16:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_7 2023-12-27T22:16:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:16:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-27T22:16:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-27T22:16:48.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:48.031+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d33040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:48.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:48.031+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997bdb900 [3/5a81880e-ca19-4e3d-9afa-fac4d233817b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_8 2023-12-27T22:16:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-12-27T22:16:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:16:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-12-27T22:16:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_8 2023-12-27T22:16:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:16:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-27T22:16:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-27T22:16:50.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:50.996+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d30680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:50.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:50.996+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997db4000 [3/d4452c8f-ad61-428d-a4d6-8e781d2d9950] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:52.889 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:16:52.888+0000 7fc355a18700 -1 reset not still connected to 0x55df116edba0 2023-12-27T22:16:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_9 2023-12-27T22:16:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-12-27T22:16:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:16:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-12-27T22:16:53.121 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3048434109452498) > weight (0.25) 2023-12-27T22:16:53.122 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9734375516979212) > weight (0.25) 2023-12-27T22:16:53.122 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-27T22:16:53.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-27T22:16:53.122 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-27T22:16:53.123 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-27T22:16:53.123 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:16:53.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:2023-12-27T22:16:53.124+0000 7fd5f7646700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 73988) UID: 1000 2023-12-27T22:16:53.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:16:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-27T22:16:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:16:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_9 2023-12-27T22:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-27T22:16:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-27T22:16:54.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:54.047+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d304e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:54.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:54.047+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997d4ac80 [3/bf528422-e47f-45b4-ada3-2e77c12c15d7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_10 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-12-27T22:16:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-27T22:16:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:16:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_10 2023-12-27T22:16:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:16:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-27T22:16:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-27T22:16:57.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:57.074+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d331e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:16:57.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:16:57.074+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997db5400 [3/ffe88c0d-d6ed-4f88-83b3-e604b69fe96e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:16:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:16:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:16:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:16:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_11 2023-12-27T22:16:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-12-27T22:16:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:16:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:16:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-12-27T22:16:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:16:59.225 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-27T22:16:59.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-27T22:16:59.225 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-27T22:16:59.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-27T22:16:59.228 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2023-12-27T22:16:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:16:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_11 2023-12-27T22:16:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:16:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:16:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:16:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_11 2023-12-27T22:16:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:16:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:16:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:16:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-12-27T22:16:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-12-27T22:17:00.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:00.087+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d33380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:00.086 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:00.087+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562993bbf180 [3/077dd2f0-f421-4104-9242-d6588dfb934d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:00.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:00.382+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56299806a9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:00.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:00.382+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562993bbf180 [3/077dd2f0-f421-4104-9242-d6588dfb934d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:00.409 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:17:00.409+0000 7f67e642e700 -1 reset not still connected to 0x55ca74ecef70 2023-12-27T22:17:00.409 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:17:00.409+0000 7f67e642e700 -1 reset not still connected to 0x55ca7795c820 2023-12-27T22:17:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_12 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-12-27T22:17:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_12 2023-12-27T22:17:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:17:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_12 2023-12-27T22:17:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:17:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-12-27T22:17:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-12-27T22:17:03.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:03.108+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d31040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:03.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:03.108+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997d4a000 [3/a041afaf-4f1b-4932-be77-439bad206770] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_13 2023-12-27T22:17:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-12-27T22:17:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:17:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-12-27T22:17:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_13 2023-12-27T22:17:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:17:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_13 2023-12-27T22:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:17:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-12-27T22:17:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-12-27T22:17:06.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:06.198+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d31520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:06.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:06.198+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997db4c80 [3/f7d5f611-7612-48c4-a55c-a9116d980d1b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_14 2023-12-27T22:17:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-12-27T22:17:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:17:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-12-27T22:17:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_14 2023-12-27T22:17:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:17:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_14 2023-12-27T22:17:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:17:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-12-27T22:17:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-12-27T22:17:09.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:09.119+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d304e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:09.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:09.119+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997d4b680 [3/cf062d67-4032-4099-9b25-373a590d58f7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_15 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-12-27T22:17:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_15 2023-12-27T22:17:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:17:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_15 2023-12-27T22:17:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:17:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-12-27T22:17:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-12-27T22:17:12.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:12.138+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d336c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:12.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:12.138+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c06500 [3/6b7b2978-4df8-43a6-826f-2914e643a2d2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_16 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-12-27T22:17:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_16 2023-12-27T22:17:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:17:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_16 2023-12-27T22:17:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:17:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-12-27T22:17:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-12-27T22:17:15.158 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:15.158+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b8b6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:15.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:15.158+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bda500 [3/299a19a7-69a7-45b4-804c-649ac34275ff] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_17 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-12-27T22:17:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_17 2023-12-27T22:17:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:17:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_17 2023-12-27T22:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-12-27T22:17:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-12-27T22:17:18.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:18.161+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d33860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:18.162 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:18.161+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997db4f00 [3/fd00cd65-5352-4569-8fe7-a9ebb5daa8a3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_18 2023-12-27T22:17:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-12-27T22:17:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:17:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-12-27T22:17:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_18 2023-12-27T22:17:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:17:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_18 2023-12-27T22:17:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:17:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-12-27T22:17:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-12-27T22:17:20.958 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:17:20.957+0000 7f67e642e700 -1 reset not still connected to 0x55ca79d921a0 2023-12-27T22:17:20.958 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:17:20.957+0000 7f67e642e700 -1 reset not still connected to 0x55ca7a473790 2023-12-27T22:17:20.959 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:17:20.958+0000 7f67e642e700 -1 reset not still connected to 0x55ca7a639860 2023-12-27T22:17:20.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:17:20.968+0000 7f43c4793700 -1 rbd::mirror::ImageReplayer: 0x55b950e97900 [3/e11e4f17-4382-4319-a42a-59efce51e962] start: already running 2023-12-27T22:17:21.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:21.001+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b8b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:21.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:21.001+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997db4500 [3/e11e4f17-4382-4319-a42a-59efce51e962] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_19 2023-12-27T22:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-12-27T22:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-12-27T22:17:23.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_19 2023-12-27T22:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_19 2023-12-27T22:17:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:17:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-12-27T22:17:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-12-27T22:17:24.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:24.017+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d31040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:24.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:24.017+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997db4780 [3/f3df9bf1-fabf-4121-bdef-2b2d3451c060] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_20 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-12-27T22:17:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_20 2023-12-27T22:17:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:17:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_20 2023-12-27T22:17:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:17:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-12-27T22:17:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-12-27T22:17:27.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:27.026+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d33a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:27.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:27.026+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c06780 [3/7149049a-7770-48be-805f-bc4a325f8c4b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_21 2023-12-27T22:17:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-12-27T22:17:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:17:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-12-27T22:17:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_21 2023-12-27T22:17:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:17:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_21 2023-12-27T22:17:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:17:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-12-27T22:17:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-12-27T22:17:30.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:30.030+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d33ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:30.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:30.030+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562993bbef00 [3/06d7ea5e-0a1f-42f1-8ff9-fb3499537169] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:30.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:30.378+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56299806fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:30.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:30.378+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562993bbef00 [3/06d7ea5e-0a1f-42f1-8ff9-fb3499537169] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_22 2023-12-27T22:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-12-27T22:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-12-27T22:17:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_22 2023-12-27T22:17:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:17:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_22 2023-12-27T22:17:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:17:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-12-27T22:17:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-12-27T22:17:33.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:33.040+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629978df380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:33.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:33.040+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bda000 [3/18ab07a2-986f-4749-9454-ce1d87f8014f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_23 2023-12-27T22:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-12-27T22:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-12-27T22:17:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_23 2023-12-27T22:17:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:17:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_23 2023-12-27T22:17:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:17:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-12-27T22:17:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-12-27T22:17:36.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:36.017+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b63ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:36.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:36.017+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997d4a280 [3/a37ade65-cdd8-4035-b15e-da4bd79e5619] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_24 2023-12-27T22:17:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-12-27T22:17:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:17:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-12-27T22:17:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_24 2023-12-27T22:17:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:17:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_24 2023-12-27T22:17:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:17:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-12-27T22:17:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-12-27T22:17:39.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:39.064+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b62ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:39.064 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:39.064+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c06a00 [3/77a94122-1c65-4804-8b45-6573f18ab770] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_25 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-12-27T22:17:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_25 2023-12-27T22:17:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:17:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_25 2023-12-27T22:17:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:17:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-12-27T22:17:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-12-27T22:17:42.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:42.280+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b631e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:42.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:42.280+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997db5180 [3/ffe3b98c-986e-4f1e-bd91-bea8441372c0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_26 2023-12-27T22:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-12-27T22:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:17:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-12-27T22:17:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_26 2023-12-27T22:17:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:17:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_26 2023-12-27T22:17:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:17:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-12-27T22:17:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-12-27T22:17:45.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:45.108+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b63380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:45.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:45.108+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997bdaf00 [3/48db3f6c-a1c5-4ad1-b8b2-0b426f65fc7a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_27 2023-12-27T22:17:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-12-27T22:17:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:17:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-12-27T22:17:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_27 2023-12-27T22:17:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:17:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_27 2023-12-27T22:17:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:17:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-12-27T22:17:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-12-27T22:17:48.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:48.100+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b62d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:48.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:48.101+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c07b80 [3/98818c22-b08f-4edd-86c0-d236cb0457ef] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_28 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-12-27T22:17:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_28 2023-12-27T22:17:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:17:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_28 2023-12-27T22:17:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:17:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-12-27T22:17:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-12-27T22:17:51.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:51.017+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629978df520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:51.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:51.018+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562993bbf900 [3/124b8604-3ea3-41f5-8bc5-de0e5cb86d1f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_29 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-12-27T22:17:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_29 2023-12-27T22:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_29 2023-12-27T22:17:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:17:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-12-27T22:17:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-12-27T22:17:54.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:54.121+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629978de000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:54.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:54.121+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997c07900 [3/916d4730-d7f2-40c8-a499-9224c313b98e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_30 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:17:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-12-27T22:17:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:17:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_30 2023-12-27T22:17:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:17:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:17:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:17:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_30 2023-12-27T22:17:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:17:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-12-27T22:17:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-12-27T22:17:57.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:57.154+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629978df1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:17:57.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:17:57.154+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bdb400 [3/505598aa-3a40-4448-9218-b902cec83230] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:17:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:17:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_31 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:17:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-12-27T22:17:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_31 2023-12-27T22:18:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:18:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_31 2023-12-27T22:18:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:18:00.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-12-27T22:18:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-12-27T22:18:00.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:00.144+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629978df6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:00.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:00.144+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562993bbf680 [3/103074c2-6a78-4270-a144-22bb49c87496] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:00.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:00.381+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d304e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:00.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:00.381+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562993bbf680 [3/103074c2-6a78-4270-a144-22bb49c87496] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_32 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-12-27T22:18:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_32 2023-12-27T22:18:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:18:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_32 2023-12-27T22:18:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:18:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-12-27T22:18:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-12-27T22:18:03.155 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:03.155+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b62680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:03.156 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:03.155+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562993bbec80 [3/02d41175-0670-4c81-82ed-381c20b18bb9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_33 2023-12-27T22:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-12-27T22:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:18:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-12-27T22:18:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_33 2023-12-27T22:18:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:18:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_33 2023-12-27T22:18:06.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:18:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-12-27T22:18:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-12-27T22:18:06.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:06.180+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997fedd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:06.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:06.180+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bda280 [3/232f1d52-1511-44c7-ac99-da55f617e6e4] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:07.197 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-27T22:18:07.197 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-27T22:18:07.197 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-27T22:18:07.197 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-27T22:18:07.197 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:18:07.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T22:18:07.198+0000 7faa35ae2700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 61990) UID: 1000 2023-12-27T22:18:07.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:18:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_34 2023-12-27T22:18:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-12-27T22:18:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:18:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-12-27T22:18:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_34 2023-12-27T22:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_34 2023-12-27T22:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-12-27T22:18:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-12-27T22:18:09.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:09.195+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b629c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:09.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:09.195+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c06000 [3/619c2405-7b4e-4332-9bd5-b1bd4115bc7d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_35 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-12-27T22:18:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_35 2023-12-27T22:18:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:18:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_35 2023-12-27T22:18:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:18:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-12-27T22:18:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-12-27T22:18:12.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:12.214+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b636c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:12.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:12.214+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997d4a780 [3/abdbf0b7-80a9-450d-bd6c-9b78f7665b62] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:18:13.299 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-27T22:18:13.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T22:18:13.300 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T22:18:13.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T22:18:13.303 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2023-12-27T22:18:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_36 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-12-27T22:18:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_36 2023-12-27T22:18:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:18:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_36 2023-12-27T22:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-12-27T22:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-12-27T22:18:15.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:15.216+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997fedba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:15.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:15.216+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997db4280 [3/d7e1cda4-b8f6-4e1c-9d85-6eaadd6f32c4] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_37 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-12-27T22:18:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:18.080 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_37 2023-12-27T22:18:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:18:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_37 2023-12-27T22:18:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:18:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-12-27T22:18:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-12-27T22:18:18.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:18.241+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997fecea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:18.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:18.241+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997d4b400 [3/cb7251f0-99f4-4c03-a067-b3a87657c534] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_38 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-12-27T22:18:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_38 2023-12-27T22:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_38 2023-12-27T22:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:18:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-12-27T22:18:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-12-27T22:18:21.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:21.015+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b63860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:21.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:21.015+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997d4a500 [3/a570017e-0f25-4b1e-bf55-91e5287082df] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_39 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:22.994 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-12-27T22:18:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_39 2023-12-27T22:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_39 2023-12-27T22:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:18:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-12-27T22:18:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-12-27T22:18:24.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:24.023+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997fed1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:24.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:24.023+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bdb180 [3/4cb400ed-6f6e-497a-ba79-d1988e890173] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_40 2023-12-27T22:18:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-12-27T22:18:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:18:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-12-27T22:18:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_40 2023-12-27T22:18:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:18:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_40 2023-12-27T22:18:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:18:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-12-27T22:18:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-12-27T22:18:27.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:27.038+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b3c340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:27.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:27.038+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562993bbf400 [3/0b9a6172-e596-4e63-ada8-38dbbc7d188d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_41 2023-12-27T22:18:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-12-27T22:18:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:18:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-12-27T22:18:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_41 2023-12-27T22:18:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:18:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_41 2023-12-27T22:18:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:18:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-12-27T22:18:29.948 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-12-27T22:18:30.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:30.060+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b3c000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:30.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:30.060+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c07680 [3/8f628a70-f2b5-4041-a2d4-bbbc883caac7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:30.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:30.377+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629978df520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:30.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:30.377+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997c07680 [3/8f628a70-f2b5-4041-a2d4-bbbc883caac7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_42 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:18:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-12-27T22:18:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_42 2023-12-27T22:18:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:18:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_42 2023-12-27T22:18:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:18:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-12-27T22:18:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-12-27T22:18:33.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:33.064+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5629978fd1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:33.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:33.064+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997c06f00 [3/7b5df11a-e491-4959-be66-344df8a2e44a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_43 2023-12-27T22:18:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-12-27T22:18:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:18:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-12-27T22:18:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_43 2023-12-27T22:18:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:18:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_43 2023-12-27T22:18:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:18:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-12-27T22:18:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-12-27T22:18:36.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:36.024+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b3dd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:36.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:36.024+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c06280 [3/6303d43d-7367-464b-9985-add1a3b57264] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_44 2023-12-27T22:18:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-12-27T22:18:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:18:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-12-27T22:18:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_44 2023-12-27T22:18:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:18:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_44 2023-12-27T22:18:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:18:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-12-27T22:18:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-12-27T22:18:39.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:39.091+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562995f31860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:39.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:39.091+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bdbb80 [3/5ce59573-6607-4c36-b8ca-ffaa5d72c073] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_45 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-12-27T22:18:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_45 2023-12-27T22:18:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:18:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_45 2023-12-27T22:18:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:18:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-12-27T22:18:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-12-27T22:18:42.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:42.098+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b3dba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:42.098 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:42.098+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997d4b180 [3/c1569f06-8c36-41d6-83d4-36589c3c6c54] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_46 2023-12-27T22:18:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-12-27T22:18:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:18:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-12-27T22:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_46 2023-12-27T22:18:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:18:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_46 2023-12-27T22:18:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:18:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-12-27T22:18:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-12-27T22:18:45.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:45.103+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562994878680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:45.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:45.103+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997c06c80 [3/79fc8fc4-a92b-41e3-8510-d70489a1905a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_47 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-12-27T22:18:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_47 2023-12-27T22:18:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:18:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_47 2023-12-27T22:18:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:18:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-12-27T22:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-12-27T22:18:48.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:48.120+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562994879380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:48.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:48.120+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997d4aa00 [3/b49aa2f8-63aa-40d1-91ed-40c882361a94] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:48.229 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.42193110045367876) > weight (0.25) 2023-12-27T22:18:48.229 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5585254208217383) > weight (0.25) 2023-12-27T22:18:48.229 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.45678986490601026) > weight (0.25) 2023-12-27T22:18:48.229 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-27T22:18:48.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-27T22:18:48.229 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-27T22:18:48.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi196.stderr:2023-12-27T22:18:48.230+0000 7f73c9718700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 53062) UID: 1000 2023-12-27T22:18:48.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:18:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_48 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-12-27T22:18:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_48 2023-12-27T22:18:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:18:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_48 2023-12-27T22:18:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:18:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-12-27T22:18:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-12-27T22:18:51.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:51.017+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b3d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:51.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:51.018+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997bdb680 [3/526175a8-e0bc-47f4-b966-8b42df9ad7e6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:51.285 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:18:51.284+0000 7f8280ae9700 -1 reset not still connected to 0x55c8e0bf96c0 2023-12-27T22:18:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_49 2023-12-27T22:18:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-12-27T22:18:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:18:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-12-27T22:18:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_49 2023-12-27T22:18:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:18:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_49 2023-12-27T22:18:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:18:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-12-27T22:18:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-12-27T22:18:54.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:54.142+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562994879ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:54.143 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:54.142+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562993bbfb80 [3/13149630-0d06-4f7b-80ad-5c081ac5165b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_50 2023-12-27T22:18:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-12-27T22:18:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:18:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local snap=snap 2023-12-27T22:18:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-12-27T22:18:56.230 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-27T22:18:56.231 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:18:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:18:56.231 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-27T22:18:56.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-27T22:18:56.231 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-27T22:18:56.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-27T22:18:56.234 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-27T22:18:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing snap: 100% complete...done. 2023-12-27T22:18:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_50 2023-12-27T22:18:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:18:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:18:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_50 2023-12-27T22:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:18:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-12-27T22:18:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-12-27T22:18:57.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:57.152+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b3c1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:18:57.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:18:57.152+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c07400 [3/8f0ce147-0f9e-4455-a07c-cb25053ca356] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:18:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-27T22:18:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ testlog 'TEST: image deletions should propagate' 2023-12-27T22:18:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee -a /tmp/tmp.yuhsqMaECB/rbd-mirror.test.log 2023-12-27T22:18:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ date '+%F %T' 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo 2023-12-27 22:18:59 TEST: image deletions should propagate 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-27 22:18:59 TEST: image deletions should propagate 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_count=0 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local count 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local last_count=0 2023-12-27T22:18:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ true 2023-12-27T22:18:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 40 2023-12-27T22:18:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in `seq 1 40` 2023-12-27T22:18:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 1 -ne 1 2023-12-27T22:18:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-27T22:18:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep 'images: ' 2023-12-27T22:18:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cut '-d ' -f 2 2023-12-27T22:18:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ count=0 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 0 = 0 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state 2023-12-27T22:18:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 40 2023-12-27T22:18:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in `seq 1 40` 2023-12-27T22:18:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test 1 -ne 1 2023-12-27T22:18:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-27T22:18:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ grep 'image health:' 2023-12-27T22:18:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cut '-d ' -f 3 2023-12-27T22:18:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: failed to query services: (13) Permission denied 2023-12-27T22:18:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ state=OK 2023-12-27T22:18:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test OK = ERROR 2023-12-27T22:18:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test OK = OK 2023-12-27T22:18:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 50 2023-12-27T22:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_1 2023-12-27T22:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-27T22:18:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_1 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_1 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-27T22:18:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-27T22:18:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:18:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:18:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_1 2023-12-27T22:18:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_1 2023-12-27T22:18:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-27T22:18:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-27T22:18:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:18:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:18:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_2 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_2 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_2 2023-12-27T22:18:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-27T22:18:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-27T22:18:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:18:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:18:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_2 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_2 2023-12-27T22:18:59.534 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-27T22:18:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-27T22:18:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:18:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:18:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_3 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_3 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_3 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-27T22:18:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-27T22:18:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:18:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:18:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:18:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:18:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_3 2023-12-27T22:18:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_3 2023-12-27T22:18:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-27T22:18:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_4 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:18:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_4 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_4 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-27T22:18:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-27T22:18:59.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:18:59.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:18:59.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:18:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-27T22:18:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:18:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_4 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_4 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-27T22:18:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-27T22:18:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:18:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_5 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_5 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_5 2023-12-27T22:18:59.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-27T22:18:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:18:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-27T22:18:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:18:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:18:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-27T22:19:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_5 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_5 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-27T22:19:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-27T22:19:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_6 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_6 2023-12-27T22:19:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_6 2023-12-27T22:19:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-27T22:19:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-27T22:19:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-27T22:19:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_6 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_6 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-27T22:19:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_7 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:19:00.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_7 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_7 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-27T22:19:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-27T22:19:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_7 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_7 2023-12-27T22:19:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-27T22:19:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_8 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:19:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_8 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_8 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-27T22:19:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-27T22:19:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-12-27T22:19:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_8 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_8 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-27T22:19:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_9 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:19:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_9 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_9 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-27T22:19:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-27T22:19:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-12-27T22:19:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_9 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_9 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_9 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-27T22:19:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_10 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_10 2023-12-27T22:19:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_10 2023-12-27T22:19:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-27T22:19:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-27T22:19:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:00.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_10 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_10 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_10 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-27T22:19:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_11 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_11 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_11 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-27T22:19:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-27T22:19:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_11 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_11 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_11 2023-12-27T22:19:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-27T22:19:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_12 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_12 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_12 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-27T22:19:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-27T22:19:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_12 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_12 2023-12-27T22:19:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_12 2023-12-27T22:19:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-27T22:19:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-27T22:19:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_13 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_13 2023-12-27T22:19:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_13 2023-12-27T22:19:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-27T22:19:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-27T22:19:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2023-12-27T22:19:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_13 2023-12-27T22:19:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_13 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_13 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-27T22:19:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-27T22:19:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_14 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_14 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_14 2023-12-27T22:19:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-27T22:19:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-27T22:19:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_14 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_14 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_14 2023-12-27T22:19:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-27T22:19:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_15 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_15 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_15 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-27T22:19:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-27T22:19:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_15 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_15 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_15 2023-12-27T22:19:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-27T22:19:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_16 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_16 2023-12-27T22:19:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_16 2023-12-27T22:19:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-27T22:19:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-27T22:19:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2023-12-27T22:19:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_16 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_16 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_16 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-27T22:19:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_17 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_17 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_17 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-27T22:19:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-27T22:19:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2023-12-27T22:19:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_17 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_17 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_17 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-27T22:19:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-27T22:19:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_18 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_18 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_18 2023-12-27T22:19:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-27T22:19:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-27T22:19:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_18 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_18 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_18 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-27T22:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_19 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:19:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_19 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_19 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-27T22:19:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-27T22:19:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2023-12-27T22:19:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_19 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_19 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_19 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-27T22:19:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_20 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_20 2023-12-27T22:19:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_20 2023-12-27T22:19:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-27T22:19:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-27T22:19:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_20 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_20 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_20 2023-12-27T22:19:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-27T22:19:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-27T22:19:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_21 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_21 2023-12-27T22:19:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_21 2023-12-27T22:19:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-27T22:19:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-27T22:19:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2023-12-27T22:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_21 2023-12-27T22:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_21 2023-12-27T22:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_21 2023-12-27T22:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-27T22:19:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-27T22:19:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_22 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_22 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_22 2023-12-27T22:19:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-27T22:19:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-27T22:19:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2023-12-27T22:19:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_22 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_22 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_22 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-27T22:19:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_23 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_23 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_23 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-27T22:19:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-27T22:19:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_23 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_23 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_23 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-27T22:19:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_24 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_24 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_24 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-27T22:19:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-27T22:19:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_24 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_24 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_24 2023-12-27T22:19:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-27T22:19:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_25 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_25 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_25 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-27T22:19:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-27T22:19:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_25 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_25 2023-12-27T22:19:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_25 2023-12-27T22:19:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-27T22:19:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-27T22:19:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_26 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_26 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_26 2023-12-27T22:19:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-27T22:19:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-27T22:19:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_26 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_26 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_26 2023-12-27T22:19:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-27T22:19:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-27T22:19:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_27 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_27 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_27 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-27T22:19:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-27T22:19:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_27 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_27 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_27 2023-12-27T22:19:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-27T22:19:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-27T22:19:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_28 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_28 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_28 2023-12-27T22:19:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-27T22:19:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-27T22:19:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2023-12-27T22:19:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_28 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_28 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_28 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-27T22:19:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-27T22:19:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_29 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_29 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_29 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-27T22:19:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-27T22:19:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_29 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_29 2023-12-27T22:19:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_29 2023-12-27T22:19:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-27T22:19:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-27T22:19:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_30 2023-12-27T22:19:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-12-27T22:19:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:19:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_30 2023-12-27T22:19:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_30 2023-12-27T22:19:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-27T22:19:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-27T22:19:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2023-12-27T22:19:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_30 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_30 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_30 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-27T22:19:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_31 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_31 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_31 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-27T22:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-27T22:19:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2023-12-27T22:19:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_31 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_31 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_31 2023-12-27T22:19:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-27T22:19:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-27T22:19:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_32 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_32 2023-12-27T22:19:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_32 2023-12-27T22:19:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-27T22:19:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-27T22:19:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2023-12-27T22:19:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_32 2023-12-27T22:19:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_32 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_32 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-27T22:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-27T22:19:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_33 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_33 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_33 2023-12-27T22:19:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-27T22:19:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-27T22:19:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2023-12-27T22:19:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_33 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_33 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_33 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-27T22:19:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-27T22:19:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_34 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_34 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_34 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-27T22:19:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-27T22:19:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_34 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_34 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_34 2023-12-27T22:19:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-27T22:19:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-27T22:19:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_35 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_35 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_35 2023-12-27T22:19:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-27T22:19:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-27T22:19:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2023-12-27T22:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_35 2023-12-27T22:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_35 2023-12-27T22:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_35 2023-12-27T22:19:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-27T22:19:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-27T22:19:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_36 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:05.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_36 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_36 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-27T22:19:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-27T22:19:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2023-12-27T22:19:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_36 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_36 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_36 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-27T22:19:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-27T22:19:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_37 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_37 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_37 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-27T22:19:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-27T22:19:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_37 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_37 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_37 2023-12-27T22:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-27T22:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-27T22:19:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_38 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_38 2023-12-27T22:19:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_38 2023-12-27T22:19:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-27T22:19:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-27T22:19:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_38 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_38 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_38 2023-12-27T22:19:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-27T22:19:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-27T22:19:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_39 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_39 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_39 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-27T22:19:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-27T22:19:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_39 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_39 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_39 2023-12-27T22:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-27T22:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-27T22:19:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_40 2023-12-27T22:19:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-12-27T22:19:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:19:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_40 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_40 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-27T22:19:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-27T22:19:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_40 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_40 2023-12-27T22:19:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_40 2023-12-27T22:19:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-27T22:19:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_41 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_41 2023-12-27T22:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_41 2023-12-27T22:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-27T22:19:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-27T22:19:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2023-12-27T22:19:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_41 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_41 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_41 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-27T22:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-27T22:19:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_42 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_42 2023-12-27T22:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_42 2023-12-27T22:19:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-27T22:19:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-27T22:19:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2023-12-27T22:19:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_42 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_42 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_42 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-27T22:19:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-27T22:19:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_43 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_43 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_43 2023-12-27T22:19:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-27T22:19:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-27T22:19:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2023-12-27T22:19:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_43 2023-12-27T22:19:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_43 2023-12-27T22:19:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_43 2023-12-27T22:19:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-27T22:19:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-27T22:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_44 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_44 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_44 2023-12-27T22:19:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-27T22:19:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-27T22:19:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_44 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_44 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_44 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-27T22:19:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_45 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_45 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_45 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-27T22:19:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-27T22:19:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_45 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_45 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_45 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-27T22:19:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_46 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:19:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_46 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_46 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-27T22:19:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:06.958 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-27T22:19:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_46 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_46 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_46 2023-12-27T22:19:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-27T22:19:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-27T22:19:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_47 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_47 2023-12-27T22:19:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_47 2023-12-27T22:19:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-27T22:19:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-27T22:19:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_47 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_47 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_47 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-27T22:19:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-27T22:19:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_48 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_48 2023-12-27T22:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_48 2023-12-27T22:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-27T22:19:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-27T22:19:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2023-12-27T22:19:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_48 2023-12-27T22:19:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_48 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_48 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-27T22:19:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_49 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_49 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_49 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-27T22:19:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-27T22:19:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2023-12-27T22:19:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_49 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_49 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_49 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-27T22:19:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_50 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_50 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_50 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-27T22:19:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-27T22:19:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2023-12-27T22:19:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_50 2023-12-27T22:19:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_50 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_50 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-27T22:19:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-27T22:19:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ testlog 'TEST: delete images during bootstrap' 2023-12-27T22:19:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ tee -a /tmp/tmp.yuhsqMaECB/rbd-mirror.test.log 2023-12-27T22:19:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ date '+%F %T' 2023-12-27T22:19:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ echo 2023-12-27 22:19:07 TEST: delete images during bootstrap 2023-12-27T22:19:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-27 22:19:07 TEST: delete images during bootstrap 2023-12-27T22:19:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-12-27T22:19:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=image 2023-12-27T22:19:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-12-27T22:19:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd: mirroring is already configured for image mode 2023-12-27T22:19:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-12-27T22:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=image 2023-12-27T22:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-12-27T22:19:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd: mirroring is already configured for image mode 2023-12-27T22:19:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ start_mirror cluster1 2023-12-27T22:19:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set_cluster_instance cluster1 cluster instance 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local val=cluster1 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster_var_name=cluster 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local instance_var_name=instance 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ cluster=cluster1 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=cluster1 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' cluster1 = cluster1 ']' 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ instance=0 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval cluster=cluster1 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cluster=cluster1 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ eval instance=0 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ instance=0 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 1 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 2023-12-27T22:19:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=test 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ seq 1 10 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_1 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_1 512M 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-12-27T22:19:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_1 2023-12-27T22:19:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:19:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-27T22:19:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:19:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_1': 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 194420782f5a 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.194420782f5a 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:19:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:19:07 2023 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:19:07 2023 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:19:07 2023 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: b19f5895-c452-4bd6-9b88-9de73f67045f 2023-12-27T22:19:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:19:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-27T22:19:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:19:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=6 2023-12-27T22:19:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:19:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:19:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 36280320 bytes 2023-12-27T22:19:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 109210624 bytes 2023-12-27T22:19:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 166558720 bytes 2023-12-27T22:19:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 250117120 bytes 2023-12-27T22:19:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 265388032 bytes 2023-12-27T22:19:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 272115712 bytes 2023-12-27T22:19:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 325378048 bytes 2023-12-27T22:19:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 426353664 bytes 2023-12-27T22:19:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 457939968 bytes 2023-12-27T22:19:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 458532864 bytes 2023-12-27T22:19:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 490557440 bytes 2023-12-27T22:19:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 550298624 bytes 2023-12-27T22:19:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 567875584 bytes 2023-12-27T22:19:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 603604992 bytes 2023-12-27T22:19:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 640272384 bytes 2023-12-27T22:19:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 659782656 bytes 2023-12-27T22:19:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 660382720 bytes 2023-12-27T22:19:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 686332928 bytes 2023-12-27T22:19:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 737776640 bytes 2023-12-27T22:19:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 768718848 bytes 2023-12-27T22:19:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 811167744 bytes 2023-12-27T22:19:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 813570048 bytes 2023-12-27T22:19:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 832011264 bytes 2023-12-27T22:19:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 843580416 bytes 2023-12-27T22:19:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 925571072 bytes 2023-12-27T22:19:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1022964736 bytes 2023-12-27T22:19:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1064713216 bytes 2023-12-27T22:19:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1145278464 bytes 2023-12-27T22:19:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1174008832 bytes 2023-12-27T22:19:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1213716480 bytes 2023-12-27T22:19:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:19:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:19:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:19:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_1 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_1 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-27T22:19:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c6e246db6a 2023-12-27T22:19:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_1 present 13c6e246db6a 2023-12-27T22:19:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c6e246db6a 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_1 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_1 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-27T22:19:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c6e246db6a 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c6e246db6a 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c6e246db6a -o 13c6e246db6a = 13c6e246db6a 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_1 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-12-27T22:19:30.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:19:30.373+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d33ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:19:30.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:19:30.373+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bdb680 [3/b19f5895-c452-4bd6-9b88-9de73f67045f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:19:45.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:19:45.064+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952779000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-12-27T22:19:45.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:19:45.064+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952779000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=ab594e73-ba6b-4c35-8043-8362c2687799, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-12-27T22:19:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring disabled 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_1 2023-12-27T22:19:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_1 2023-12-27T22:19:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-27T22:19:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-27T22:19:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:19:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-27T22:19:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_1 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_1 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-27T22:19:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ purge_snapshots cluster2 mirror image_1 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-12-27T22:19:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-27T22:19:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:19:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:19:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_1 2023-12-27T22:19:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_1 2023-12-27T22:19:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-27T22:19:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-27T22:19:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:19:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:19:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_2 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_2 512M 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:19:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-12-27T22:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_2 2023-12-27T22:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-27T22:19:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:19:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-27T22:19:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_2': 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1959ec59e68c 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1959ec59e68c 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:19:45 2023 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:19:45 2023 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:19:45 2023 2023-12-27T22:19:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:19:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:19:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 30f8eae4-46af-4168-a4cc-195469b5b994 2023-12-27T22:19:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:19:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-27T22:19:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:19:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:19:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=9 2023-12-27T22:19:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:19:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:19:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 53695488 bytes 2023-12-27T22:19:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 54300672 bytes 2023-12-27T22:19:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 66072576 bytes 2023-12-27T22:19:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 66686976 bytes 2023-12-27T22:19:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 90063872 bytes 2023-12-27T22:19:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 101117952 bytes 2023-12-27T22:19:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 185927680 bytes 2023-12-27T22:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 243937280 bytes 2023-12-27T22:19:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 288176128 bytes 2023-12-27T22:19:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 349212672 bytes 2023-12-27T22:19:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 375571456 bytes 2023-12-27T22:19:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 417301504 bytes 2023-12-27T22:19:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 421686272 bytes 2023-12-27T22:19:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 440301568 bytes 2023-12-27T22:19:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 457268224 bytes 2023-12-27T22:19:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 535602176 bytes 2023-12-27T22:19:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 576103424 bytes 2023-12-27T22:19:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 589997056 bytes 2023-12-27T22:19:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 645583872 bytes 2023-12-27T22:19:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 698545152 bytes 2023-12-27T22:19:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 755243008 bytes 2023-12-27T22:19:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 843947008 bytes 2023-12-27T22:19:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 962143232 bytes 2023-12-27T22:19:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1037713408 bytes 2023-12-27T22:19:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1042384896 bytes 2023-12-27T22:19:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1153197056 bytes 2023-12-27T22:19:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1207910400 bytes 2023-12-27T22:19:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1265809408 bytes 2023-12-27T22:19:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1347581952 bytes 2023-12-27T22:19:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1360607232 bytes 2023-12-27T22:19:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1425971200 bytes 2023-12-27T22:19:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1522272256 bytes 2023-12-27T22:19:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1531936768 bytes 2023-12-27T22:19:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1591290880 bytes 2023-12-27T22:19:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1635181568 bytes 2023-12-27T22:19:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1635777536 bytes 2023-12-27T22:19:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1686972416 bytes 2023-12-27T22:19:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1760125952 bytes 2023-12-27T22:19:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1801689088 bytes 2023-12-27T22:19:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1849790464 bytes 2023-12-27T22:19:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1857356800 bytes 2023-12-27T22:19:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1885596672 bytes 2023-12-27T22:19:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1982921728 bytes 2023-12-27T22:19:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1990322176 bytes 2023-12-27T22:19:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:19:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:19:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:19:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-12-27T22:19:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_2 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_2 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-27T22:19:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c6d21976b7 2023-12-27T22:19:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:19:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:19:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_2 present 13c6d21976b7 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c6d21976b7 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_2 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_2 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-27T22:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:19:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c6d21976b7 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c6d21976b7 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c6d21976b7 -o 13c6d21976b7 = 13c6d21976b7 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_2 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:19:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-12-27T22:20:00.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:20:00.374+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b63860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:20:00.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:20:00.374+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997c07400 [3/30f8eae4-46af-4168-a4cc-195469b5b994] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:20:02.303 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-27T22:20:02.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-27T22:20:02.304 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-27T22:20:02.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:2023-12-27T22:20:02.306+0000 7f0464e57700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 77634) UID: 1000 2023-12-27T22:20:02.506 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:20:11.305 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-27T22:20:11.305 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:20:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:20:11.306 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-27T22:20:11.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-27T22:20:11.306 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-27T22:20:11.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-27T22:20:11.309 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2023-12-27T22:20:26.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:20:26.150+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b950efec00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-12-27T22:20:26.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:20:26.150+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b950efec00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=ab594e73-ba6b-4c35-8043-8362c2687799, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-12-27T22:20:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring disabled 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:20:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:20:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_2 2023-12-27T22:20:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:20:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:20:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_2 2023-12-27T22:20:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-27T22:20:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:20:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-27T22:20:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:20:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:20:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_2 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:20:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:20:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_2 2023-12-27T22:20:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-27T22:20:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:20:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-27T22:20:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:20:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:20:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:20:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ purge_snapshots cluster2 mirror image_2 2023-12-27T22:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:20:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-12-27T22:20:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-27T22:20:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:20:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:20:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:20:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:20:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_2 2023-12-27T22:20:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:20:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_2 2023-12-27T22:20:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-27T22:20:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-27T22:20:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:20:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:20:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:20:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_3 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_3 512M 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:20:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-12-27T22:20:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_3 2023-12-27T22:20:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:20:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:20:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:20:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-27T22:20:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:20:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_3': 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 19757d8c23b7 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.19757d8c23b7 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:20:26 2023 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:20:26 2023 2023-12-27T22:20:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:20:26 2023 2023-12-27T22:20:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:20:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:20:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 96f97c57-8fe9-4db0-bbb5-6b869438feca 2023-12-27T22:20:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:20:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-27T22:20:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:20:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:20:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:20:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=10 2023-12-27T22:20:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:20:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:20:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 55123968 bytes 2023-12-27T22:20:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 84672512 bytes 2023-12-27T22:20:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 122094592 bytes 2023-12-27T22:20:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 192103424 bytes 2023-12-27T22:20:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 216503296 bytes 2023-12-27T22:20:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 265747456 bytes 2023-12-27T22:20:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 322596864 bytes 2023-12-27T22:20:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 384825344 bytes 2023-12-27T22:20:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 439518208 bytes 2023-12-27T22:20:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 468980736 bytes 2023-12-27T22:20:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 469596160 bytes 2023-12-27T22:20:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 525332480 bytes 2023-12-27T22:20:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 638273536 bytes 2023-12-27T22:20:30.435 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T22:20:30.434+0000 7f7842c55700 -1 bdev(0x55c83edee400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-27T22:20:30.436 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7842c55700 time 2023-12-27T22:20:30.436291+0000 2023-12-27T22:20:30.436 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:20:30.436 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55c833702068] 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: clone() 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T22:20:30.436+0000 7f7842c55700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7842c55700 time 2023-12-27T22:20:30.436291+0000 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55c833702068] 2023-12-27T22:20:30.437 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: clone() 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:*** Caught signal (Aborted) ** 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: in thread 7f7842c55700 thread_name:bstore_aio 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.438 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f784eac8ce0] 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: gsignal() 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: abort() 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55c833702139] 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 8: clone() 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T22:20:30.438+0000 7f7842c55700 -1 *** Caught signal (Aborted) ** 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: in thread 7f7842c55700 thread_name:bstore_aio 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f784eac8ce0] 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: gsignal() 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: abort() 2023-12-27T22:20:30.439 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55c833702139] 2023-12-27T22:20:30.440 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.440 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.440 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.440 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 8: clone() 2023-12-27T22:20:30.440 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:20:30.440 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 685443072 bytes 2023-12-27T22:20:30.457 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: -23> 2023-12-27T22:20:30.434+0000 7f7842c55700 -1 bdev(0x55c83edee400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-27T22:20:30.457 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: -14> 2023-12-27T22:20:30.436+0000 7f7842c55700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7842c55700 time 2023-12-27T22:20:30.436291+0000 2023-12-27T22:20:30.457 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:20:30.457 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.457 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.457 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55c833702068] 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: clone() 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 0> 2023-12-27T22:20:30.438+0000 7f7842c55700 -1 *** Caught signal (Aborted) ** 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: in thread 7f7842c55700 thread_name:bstore_aio 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f784eac8ce0] 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: gsignal() 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: abort() 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55c833702139] 2023-12-27T22:20:30.458 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.459 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.459 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.459 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 8: clone() 2023-12-27T22:20:30.459 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:20:30.459 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.459 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:2023-12-27T22:20:30.447+0000 7f783ba38700 -1 bdev(0x55c83edeec00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: -715> 2023-12-27T22:20:30.434+0000 7f7842c55700 -1 bdev(0x55c83edee400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: -706> 2023-12-27T22:20:30.436+0000 7f7842c55700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7842c55700 time 2023-12-27T22:20:30.436291+0000 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55c833702068] 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.468 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: clone() 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: -692> 2023-12-27T22:20:30.438+0000 7f7842c55700 -1 *** Caught signal (Aborted) ** 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: in thread 7f7842c55700 thread_name:bstore_aio 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f784eac8ce0] 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2: gsignal() 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 3: abort() 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55c833702139] 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55c83426f964] 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55c83427ae01] 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f784eabe1cf] 2023-12-27T22:20:30.469 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 8: clone() 2023-12-27T22:20:30.470 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:20:30.470 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: 2023-12-27T22:20:30.470 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr: -108> 2023-12-27T22:20:30.447+0000 7f783ba38700 -1 bdev(0x55c83edeec00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-27T22:20:30.631 INFO:tasks.ceph.cluster2.osd.0.smithi196.stderr:daemon-helper: command failed with exit status 5 2023-12-27T22:20:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 818042880 bytes 2023-12-27T22:20:32.235 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.761728136809297) > weight (0.25) 2023-12-27T22:20:32.235 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-27T22:20:32.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-27T22:20:32.236 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-27T22:20:32.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi196.stderr:2023-12-27T22:20:32.242+0000 7f88b25bd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 61961) UID: 1000 2023-12-27T22:20:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 866769920 bytes 2023-12-27T22:20:32.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-27T22:20:32.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:20:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 932542464 bytes 2023-12-27T22:20:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 968301568 bytes 2023-12-27T22:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 994887680 bytes 2023-12-27T22:20:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1045922816 bytes 2023-12-27T22:20:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1059444736 bytes 2023-12-27T22:20:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1087542272 bytes 2023-12-27T22:20:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1146087424 bytes 2023-12-27T22:20:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1208908800 bytes 2023-12-27T22:20:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1239245824 bytes 2023-12-27T22:20:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1250371584 bytes 2023-12-27T22:20:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1311988736 bytes 2023-12-27T22:20:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1370800128 bytes 2023-12-27T22:20:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1402375168 bytes 2023-12-27T22:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1425083392 bytes 2023-12-27T22:20:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1433015296 bytes 2023-12-27T22:20:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1482953728 bytes 2023-12-27T22:20:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1519646720 bytes 2023-12-27T22:20:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1549640704 bytes 2023-12-27T22:20:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1575727104 bytes 2023-12-27T22:20:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1652968448 bytes 2023-12-27T22:20:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1674939392 bytes 2023-12-27T22:20:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1699115008 bytes 2023-12-27T22:20:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1699710976 bytes 2023-12-27T22:20:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1750952960 bytes 2023-12-27T22:20:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1783498752 bytes 2023-12-27T22:20:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1864589312 bytes 2023-12-27T22:20:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1868950528 bytes 2023-12-27T22:20:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1968544768 bytes 2023-12-27T22:20:36.236 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-27T22:20:36.236 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:20:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:20:36.237 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-27T22:20:36.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-27T22:20:36.237 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-27T22:20:36.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-27T22:20:36.241 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-12-27T22:20:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2030737408 bytes 2023-12-27T22:20:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2043315200 bytes 2023-12-27T22:20:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2071992320 bytes 2023-12-27T22:20:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2085881856 bytes 2023-12-27T22:20:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2107860992 bytes 2023-12-27T22:20:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2128640000 bytes 2023-12-27T22:20:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2170103808 bytes 2023-12-27T22:20:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2170722304 bytes 2023-12-27T22:20:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:20:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:20:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_3 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_3 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-27T22:20:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:20:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c6bfb6c5d7 2023-12-27T22:20:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:20:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:20:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_3 present 13c6bfb6c5d7 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c6bfb6c5d7 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_3 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_3 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-27T22:20:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:20:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c6bfb6c5d7 2023-12-27T22:20:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c6bfb6c5d7 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c6bfb6c5d7 -o 13c6bfb6c5d7 = 13c6bfb6c5d7 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_3 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:20:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-12-27T22:20:37.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-27T22:20:43.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-27T22:20:48.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-27T22:20:53.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-27T22:20:59.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-27T22:21:00.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:21:00.374+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562996084b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:21:00.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:21:00.374+0000 7ff82f823700 -1 rbd::mirror::ImageReplayer: 0x562997bdb680 [3/96f97c57-8fe9-4db0-bbb5-6b869438feca] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:21:04.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-27T22:21:09.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:21:09.000+0000 7f43c4793700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952880800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-12-27T22:21:09.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:21:09.000+0000 7f43c4793700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952880800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=ab594e73-ba6b-4c35-8043-8362c2687799, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-12-27T22:21:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring disabled 2023-12-27T22:21:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-27T22:21:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:21:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:21:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:21:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:21:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:21:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:21:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_3 2023-12-27T22:21:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:21:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:21:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_3 2023-12-27T22:21:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-27T22:21:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:21:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-27T22:21:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:21:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:21:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:21:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_3 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_3 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-27T22:21:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:21:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ purge_snapshots cluster2 mirror image_3 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:21:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-12-27T22:21:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-27T22:21:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:21:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:21:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:21:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_3 2023-12-27T22:21:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_3 2023-12-27T22:21:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-27T22:21:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-27T22:21:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:21:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:21:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:21:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_4 2023-12-27T22:21:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_4 512M 2023-12-27T22:21:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-12-27T22:21:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_4 2023-12-27T22:21:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:21:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-27T22:21:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:21:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-27T22:21:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_4': 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 1993142374b 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.1993142374b 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:21:09 2023 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:21:09 2023 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:21:09 2023 2023-12-27T22:21:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:21:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:21:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: f4d5829e-9392-47d8-946c-6ba699de69a0 2023-12-27T22:21:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:21:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-27T22:21:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:21:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=13 2023-12-27T22:21:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:21:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:21:10.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-27T22:21:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 32216064 bytes 2023-12-27T22:21:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 67919872 bytes 2023-12-27T22:21:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 125585408 bytes 2023-12-27T22:21:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 154272768 bytes 2023-12-27T22:21:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 210230272 bytes 2023-12-27T22:21:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 283354112 bytes 2023-12-27T22:21:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 312976384 bytes 2023-12-27T22:21:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 321192960 bytes 2023-12-27T22:21:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 377274368 bytes 2023-12-27T22:21:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 403041280 bytes 2023-12-27T22:21:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 434264064 bytes 2023-12-27T22:21:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 525743104 bytes 2023-12-27T22:21:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 556225536 bytes 2023-12-27T22:21:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 595766272 bytes 2023-12-27T22:21:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 626171904 bytes 2023-12-27T22:21:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 655975424 bytes 2023-12-27T22:21:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 713299968 bytes 2023-12-27T22:21:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 774952960 bytes 2023-12-27T22:21:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 832337920 bytes 2023-12-27T22:21:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 852713472 bytes 2023-12-27T22:21:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 867923968 bytes 2023-12-27T22:21:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 920294400 bytes 2023-12-27T22:21:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 984293376 bytes 2023-12-27T22:21:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1025974272 bytes 2023-12-27T22:21:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1034045440 bytes 2023-12-27T22:21:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1064839168 bytes 2023-12-27T22:21:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1122762752 bytes 2023-12-27T22:21:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1165849600 bytes 2023-12-27T22:21:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1197145088 bytes 2023-12-27T22:21:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1257833472 bytes 2023-12-27T22:21:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1259026432 bytes 2023-12-27T22:21:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1364670464 bytes 2023-12-27T22:21:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1371960320 bytes 2023-12-27T22:21:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1385602048 bytes 2023-12-27T22:21:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1405990912 bytes 2023-12-27T22:21:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1451544576 bytes 2023-12-27T22:21:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1518192640 bytes 2023-12-27T22:21:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1602463744 bytes 2023-12-27T22:21:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1652645888 bytes 2023-12-27T22:21:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1709963264 bytes 2023-12-27T22:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1749999616 bytes 2023-12-27T22:21:15.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-27T22:21:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1792788480 bytes 2023-12-27T22:21:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1814574080 bytes 2023-12-27T22:21:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1830060032 bytes 2023-12-27T22:21:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1877427200 bytes 2023-12-27T22:21:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1912724480 bytes 2023-12-27T22:21:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1915008000 bytes 2023-12-27T22:21:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1968611328 bytes 2023-12-27T22:21:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2030821376 bytes 2023-12-27T22:21:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2102215680 bytes 2023-12-27T22:21:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2126640128 bytes 2023-12-27T22:21:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2212464640 bytes 2023-12-27T22:21:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2248909824 bytes 2023-12-27T22:21:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2275275776 bytes 2023-12-27T22:21:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2332142592 bytes 2023-12-27T22:21:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2356190208 bytes 2023-12-27T22:21:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2382806016 bytes 2023-12-27T22:21:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2391401472 bytes 2023-12-27T22:21:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2463002624 bytes 2023-12-27T22:21:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2519498752 bytes 2023-12-27T22:21:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2653639680 bytes 2023-12-27T22:21:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2670871552 bytes 2023-12-27T22:21:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2687956992 bytes 2023-12-27T22:21:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2745156608 bytes 2023-12-27T22:21:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2745787392 bytes 2023-12-27T22:21:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2777248768 bytes 2023-12-27T22:21:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2899297280 bytes 2023-12-27T22:21:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2924305408 bytes 2023-12-27T22:21:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2949285888 bytes 2023-12-27T22:21:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 2998283264 bytes 2023-12-27T22:21:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3040334848 bytes 2023-12-27T22:21:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3183872000 bytes 2023-12-27T22:21:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3211035648 bytes 2023-12-27T22:21:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3262953472 bytes 2023-12-27T22:21:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3271895040 bytes 2023-12-27T22:21:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3272487936 bytes 2023-12-27T22:21:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3312281600 bytes 2023-12-27T22:21:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3322863616 bytes 2023-12-27T22:21:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3343389696 bytes 2023-12-27T22:21:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3413325824 bytes 2023-12-27T22:21:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8100 IOs, 3420817408 bytes 2023-12-27T22:21:21.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-27T22:21:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8200 IOs, 3476171776 bytes 2023-12-27T22:21:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8300 IOs, 3479390208 bytes 2023-12-27T22:21:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8400 IOs, 3587516416 bytes 2023-12-27T22:21:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8500 IOs, 3588272128 bytes 2023-12-27T22:21:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8600 IOs, 3599490048 bytes 2023-12-27T22:21:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8700 IOs, 3624328192 bytes 2023-12-27T22:21:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8800 IOs, 3624961024 bytes 2023-12-27T22:21:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8900 IOs, 3638307840 bytes 2023-12-27T22:21:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9000 IOs, 3645828096 bytes 2023-12-27T22:21:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9100 IOs, 3670988800 bytes 2023-12-27T22:21:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9200 IOs, 3762562048 bytes 2023-12-27T22:21:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9300 IOs, 3787593728 bytes 2023-12-27T22:21:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9400 IOs, 3865098240 bytes 2023-12-27T22:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9500 IOs, 3926875136 bytes 2023-12-27T22:21:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9600 IOs, 3956541440 bytes 2023-12-27T22:21:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9700 IOs, 4048590848 bytes 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:21:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_4 2023-12-27T22:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_4 2023-12-27T22:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-27T22:21:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:21:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c64b1f52da 2023-12-27T22:21:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:21:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_4 present 13c64b1f52da 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c64b1f52da 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_4 2023-12-27T22:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_4 2023-12-27T22:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-27T22:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:21:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c64b1f52da 2023-12-27T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c64b1f52da 2023-12-27T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c64b1f52da -o 13c64b1f52da = 13c64b1f52da 2023-12-27T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_4 2023-12-27T22:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-12-27T22:21:26.407 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-27T22:21:30.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:21:30.375+0000 7ff828815700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997d31520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:21:30.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:21:30.375+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997bdb680 [3/f4d5829e-9392-47d8-946c-6ba699de69a0] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:21:31.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-27T22:21:37.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-27T22:21:42.619 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-27T22:21:45.310 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.926467642169303) > weight (0.25) 2023-12-27T22:21:45.310 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6465947481297187) > weight (0.25) 2023-12-27T22:21:45.311 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5892968965454057) > weight (0.25) 2023-12-27T22:21:45.311 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5834843949578971) > weight (0.25) 2023-12-27T22:21:45.311 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-27T22:21:48.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-27T22:21:52.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:21:52.146+0000 7f43c4793700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b9555f3c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-12-27T22:21:52.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:21:52.146+0000 7f43c4793700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b9555f3c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=ab594e73-ba6b-4c35-8043-8362c2687799, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-12-27T22:21:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring disabled 2023-12-27T22:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-27T22:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_4 2023-12-27T22:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_4 2023-12-27T22:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-27T22:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:21:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-27T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:21:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:21:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_4 2023-12-27T22:21:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:21:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:21:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_4 2023-12-27T22:21:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-27T22:21:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:21:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-27T22:21:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:21:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:21:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:21:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:21:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ purge_snapshots cluster2 mirror image_4 2023-12-27T22:21:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-12-27T22:21:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-27T22:21:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:21:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_4 2023-12-27T22:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_4 2023-12-27T22:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-27T22:21:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-27T22:21:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:21:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_5 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_5 512M 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:21:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:21:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-12-27T22:21:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_5 2023-12-27T22:21:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:21:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:21:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-27T22:21:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:21:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-27T22:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_5': 2023-12-27T22:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 19ab12fff9eb 2023-12-27T22:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.19ab12fff9eb 2023-12-27T22:21:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:21:52 2023 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:21:52 2023 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:21:52 2023 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 10657066-d34d-4ae0-996d-87cf80955429 2023-12-27T22:21:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:21:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-27T22:21:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:21:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:21:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:21:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:21:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=8 2023-12-27T22:21:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:21:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:21:53.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-27T22:21:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 40037376 bytes 2023-12-27T22:21:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 62835712 bytes 2023-12-27T22:21:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 125126656 bytes 2023-12-27T22:21:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 148500480 bytes 2023-12-27T22:21:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 205163520 bytes 2023-12-27T22:21:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 235177984 bytes 2023-12-27T22:21:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 284093440 bytes 2023-12-27T22:21:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 326096896 bytes 2023-12-27T22:21:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 348733440 bytes 2023-12-27T22:21:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 386106368 bytes 2023-12-27T22:21:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 451511296 bytes 2023-12-27T22:21:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 478764032 bytes 2023-12-27T22:21:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 525696000 bytes 2023-12-27T22:21:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 542602240 bytes 2023-12-27T22:21:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 567481344 bytes 2023-12-27T22:21:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 582961152 bytes 2023-12-27T22:21:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 584095744 bytes 2023-12-27T22:21:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 666510336 bytes 2023-12-27T22:21:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 707778560 bytes 2023-12-27T22:21:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 731010048 bytes 2023-12-27T22:21:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 783813632 bytes 2023-12-27T22:21:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 830006272 bytes 2023-12-27T22:21:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 859180032 bytes 2023-12-27T22:21:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 887131136 bytes 2023-12-27T22:21:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 902275072 bytes 2023-12-27T22:21:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 972222464 bytes 2023-12-27T22:21:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 984359936 bytes 2023-12-27T22:21:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1035892736 bytes 2023-12-27T22:21:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1064912896 bytes 2023-12-27T22:21:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1105378304 bytes 2023-12-27T22:21:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1161603072 bytes 2023-12-27T22:21:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1218860032 bytes 2023-12-27T22:21:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1231722496 bytes 2023-12-27T22:21:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1252161536 bytes 2023-12-27T22:21:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1252765696 bytes 2023-12-27T22:21:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1277554688 bytes 2023-12-27T22:21:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1398368256 bytes 2023-12-27T22:21:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1424889856 bytes 2023-12-27T22:21:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1483281408 bytes 2023-12-27T22:21:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1510349824 bytes 2023-12-27T22:21:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1532068864 bytes 2023-12-27T22:21:58.830 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-27T22:21:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1532664832 bytes 2023-12-27T22:21:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1577694208 bytes 2023-12-27T22:21:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1658767360 bytes 2023-12-27T22:21:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1726508032 bytes 2023-12-27T22:21:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1752549376 bytes 2023-12-27T22:21:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1838358528 bytes 2023-12-27T22:21:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1887938560 bytes 2023-12-27T22:22:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1905725440 bytes 2023-12-27T22:22:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1906340864 bytes 2023-12-27T22:22:00.242 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3747929642563361) > weight (0.25) 2023-12-27T22:22:00.242 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.33822619186864966) > weight (0.25) 2023-12-27T22:22:00.242 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9736678925204838) > weight (0.25) 2023-12-27T22:22:00.242 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-27T22:22:00.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-27T22:22:00.243 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-27T22:22:00.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi196.stderr:2023-12-27T22:22:00.243+0000 7ff992db9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 78640) UID: 1000 2023-12-27T22:22:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 1977839616 bytes 2023-12-27T22:22:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2015785984 bytes 2023-12-27T22:22:00.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:22:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2016415744 bytes 2023-12-27T22:22:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2044069888 bytes 2023-12-27T22:22:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2121143296 bytes 2023-12-27T22:22:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2145826816 bytes 2023-12-27T22:22:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2174765056 bytes 2023-12-27T22:22:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2247181312 bytes 2023-12-27T22:22:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:22:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:22:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:22:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:22:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_5 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:22:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:22:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_5 2023-12-27T22:22:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-27T22:22:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:22:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c64bb4eea7 2023-12-27T22:22:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:22:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_5 present 13c64bb4eea7 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c64bb4eea7 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_5 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:22:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_5 2023-12-27T22:22:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-27T22:22:01.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:22:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c64bb4eea7 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c64bb4eea7 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c64bb4eea7 -o 13c64bb4eea7 = 13c64bb4eea7 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_5 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-12-27T22:22:04.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-27T22:22:07.244 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-27T22:22:07.244 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:22:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:22:07.245 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-27T22:22:07.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-27T22:22:07.245 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-27T22:22:07.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-27T22:22:07.248 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-12-27T22:22:09.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-27T22:22:15.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-27T22:22:20.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-27T22:22:25.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-27T22:22:30.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:22:30.374+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997fed520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:22:30.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:22:30.374+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997c07400 [3/10657066-d34d-4ae0-996d-87cf80955429] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:22:31.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-27T22:22:32.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:22:32.255+0000 7f43c4793700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952b3bc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-12-27T22:22:32.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:22:32.255+0000 7f43c4793700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952b3bc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=ab594e73-ba6b-4c35-8043-8362c2687799, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-12-27T22:22:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring disabled 2023-12-27T22:22:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_5 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_5 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-27T22:22:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:22:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-27T22:22:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:22:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:22:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:22:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_5 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_5 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-27T22:22:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:22:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ purge_snapshots cluster2 mirror image_5 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-12-27T22:22:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-27T22:22:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:22:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:22:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_5 2023-12-27T22:22:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_5 2023-12-27T22:22:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-27T22:22:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-27T22:22:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_6 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_6 512M 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:22:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:22:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-12-27T22:22:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_6 2023-12-27T22:22:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:22:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:22:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-27T22:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:22:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_6': 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 19c6c2dbcfb 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.19c6c2dbcfb 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:22:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:22:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:22:32 2023 2023-12-27T22:22:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:22:32 2023 2023-12-27T22:22:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:22:32 2023 2023-12-27T22:22:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:22:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:22:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 3f21046a-0ccb-483b-8f31-47b4c00bdbc1 2023-12-27T22:22:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:22:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-27T22:22:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:22:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:22:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=13 2023-12-27T22:22:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:22:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:22:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 5745664 bytes 2023-12-27T22:22:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 79816704 bytes 2023-12-27T22:22:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 150501376 bytes 2023-12-27T22:22:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 186534912 bytes 2023-12-27T22:22:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 211288064 bytes 2023-12-27T22:22:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 276188160 bytes 2023-12-27T22:22:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 310480896 bytes 2023-12-27T22:22:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 408564736 bytes 2023-12-27T22:22:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 460377088 bytes 2023-12-27T22:22:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 514110464 bytes 2023-12-27T22:22:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 624869376 bytes 2023-12-27T22:22:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 664367104 bytes 2023-12-27T22:22:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 729404416 bytes 2023-12-27T22:22:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 795151360 bytes 2023-12-27T22:22:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 840167424 bytes 2023-12-27T22:22:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 894520320 bytes 2023-12-27T22:22:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 956556288 bytes 2023-12-27T22:22:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 968666112 bytes 2023-12-27T22:22:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 1017899008 bytes 2023-12-27T22:22:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 1102754816 bytes 2023-12-27T22:22:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 1153785856 bytes 2023-12-27T22:22:36.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-27T22:22:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 1178274816 bytes 2023-12-27T22:22:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 1267468288 bytes 2023-12-27T22:22:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 1308850176 bytes 2023-12-27T22:22:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 1319636992 bytes 2023-12-27T22:22:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1350603776 bytes 2023-12-27T22:22:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1372145664 bytes 2023-12-27T22:22:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1399724032 bytes 2023-12-27T22:22:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1420822528 bytes 2023-12-27T22:22:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1449403392 bytes 2023-12-27T22:22:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1496114176 bytes 2023-12-27T22:22:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1535852544 bytes 2023-12-27T22:22:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1536492544 bytes 2023-12-27T22:22:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1615882240 bytes 2023-12-27T22:22:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1675513856 bytes 2023-12-27T22:22:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1707130880 bytes 2023-12-27T22:22:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1737436160 bytes 2023-12-27T22:22:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1774245888 bytes 2023-12-27T22:22:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1826695168 bytes 2023-12-27T22:22:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1849832448 bytes 2023-12-27T22:22:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1850452992 bytes 2023-12-27T22:22:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1870835712 bytes 2023-12-27T22:22:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1927501824 bytes 2023-12-27T22:22:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1977370624 bytes 2023-12-27T22:22:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 2025492480 bytes 2023-12-27T22:22:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 2042092544 bytes 2023-12-27T22:22:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 2073133056 bytes 2023-12-27T22:22:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 2118791168 bytes 2023-12-27T22:22:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 2128122880 bytes 2023-12-27T22:22:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2237229056 bytes 2023-12-27T22:22:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2330931200 bytes 2023-12-27T22:22:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2380726272 bytes 2023-12-27T22:22:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2420397056 bytes 2023-12-27T22:22:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2476150784 bytes 2023-12-27T22:22:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2530957312 bytes 2023-12-27T22:22:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2617736192 bytes 2023-12-27T22:22:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2675922944 bytes 2023-12-27T22:22:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2767253504 bytes 2023-12-27T22:22:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2863671296 bytes 2023-12-27T22:22:42.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-27T22:22:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2899931136 bytes 2023-12-27T22:22:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2991114240 bytes 2023-12-27T22:22:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 3060348928 bytes 2023-12-27T22:22:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 3082040320 bytes 2023-12-27T22:22:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 3093132288 bytes 2023-12-27T22:22:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 3093769216 bytes 2023-12-27T22:22:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 3146793984 bytes 2023-12-27T22:22:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 3194731520 bytes 2023-12-27T22:22:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 3202274304 bytes 2023-12-27T22:22:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 3238025216 bytes 2023-12-27T22:22:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 3271669760 bytes 2023-12-27T22:22:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 3276932096 bytes 2023-12-27T22:22:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 3284685824 bytes 2023-12-27T22:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 3342732288 bytes 2023-12-27T22:22:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 3385534464 bytes 2023-12-27T22:22:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3465379840 bytes 2023-12-27T22:22:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3495476224 bytes 2023-12-27T22:22:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3535782912 bytes 2023-12-27T22:22:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3573904384 bytes 2023-12-27T22:22:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3659872256 bytes 2023-12-27T22:22:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3713847296 bytes 2023-12-27T22:22:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8100 IOs, 3749081088 bytes 2023-12-27T22:22:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8200 IOs, 3798832128 bytes 2023-12-27T22:22:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8300 IOs, 3831069696 bytes 2023-12-27T22:22:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8400 IOs, 3870181376 bytes 2023-12-27T22:22:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8500 IOs, 3905906688 bytes 2023-12-27T22:22:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8600 IOs, 3916931072 bytes 2023-12-27T22:22:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8700 IOs, 4002954240 bytes 2023-12-27T22:22:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8800 IOs, 4056500224 bytes 2023-12-27T22:22:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8900 IOs, 4070431744 bytes 2023-12-27T22:22:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9000 IOs, 4105906176 bytes 2023-12-27T22:22:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9100 IOs, 4128142336 bytes 2023-12-27T22:22:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:9200 IOs, 4185833472 bytes 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:22:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_6 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_6 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-27T22:22:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:22:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c6b67a53f4 2023-12-27T22:22:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:22:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_6 present 13c6b67a53f4 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c6b67a53f4 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_6 2023-12-27T22:22:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:22:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:22:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_6 2023-12-27T22:22:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-27T22:22:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:22:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c6b67a53f4 2023-12-27T22:22:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c6b67a53f4 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c6b67a53f4 -o 13c6b67a53f4 = 13c6b67a53f4 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_6 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:22:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-12-27T22:22:47.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-27T22:22:52.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~140s 2023-12-27T22:22:58.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-27T22:23:00.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:23:00.375+0000 7ff829016700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562997b561a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:23:00.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:23:00.375+0000 7ff82f022700 -1 rbd::mirror::ImageReplayer: 0x562997d4b900 [3/3f21046a-0ccb-483b-8f31-47b4c00bdbc1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:23:03.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-27T22:23:09.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-27T22:23:14.311 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5548463657381852) > weight (0.25) 2023-12-27T22:23:14.311 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-27T22:23:14.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-27T22:23:14.312 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-27T22:23:14.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:2023-12-27T22:23:14.314+0000 7ff830024700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 59880) UID: 1000 2023-12-27T22:23:14.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:23:14.426+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b950c29c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-12-27T22:23:14.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:23:14.429+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b950c29c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-12-27T22:23:14.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:23:14.429+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b950c29c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=ab594e73-ba6b-4c35-8043-8362c2687799, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-12-27T22:23:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring disabled 2023-12-27T22:23:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-27T22:23:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:23:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:23:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:23:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_6 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_6 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-27T22:23:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:23:14.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi196.stderr:2023-12-27T22:23:14.470+0000 7f68e1af7700 -1 rbd::mirror::ImageMap: 0x55814c6bf0e0 handle_peer_ack_remove: failed to remove global_image_id=3f21046a-0ccb-483b-8f31-47b4c00bdbc1 2023-12-27T22:23:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-27T22:23:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:23:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:23:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:23:14.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-27T22:23:14.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_6 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:23:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:23:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_6 2023-12-27T22:23:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-27T22:23:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:23:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-27T22:23:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:23:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:23:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:23:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:23:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ purge_snapshots cluster2 mirror image_6 2023-12-27T22:23:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:23:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-12-27T22:23:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-27T22:23:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:23:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:23:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:23:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_6 2023-12-27T22:23:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_6 2023-12-27T22:23:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-27T22:23:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-27T22:23:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:23:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:23:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:23:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_7 2023-12-27T22:23:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_7 512M 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-12-27T22:23:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_7 2023-12-27T22:23:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:23:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-27T22:23:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:23:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_7': 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 19d7a4390273 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.19d7a4390273 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:23:14 2023 2023-12-27T22:23:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:23:14 2023 2023-12-27T22:23:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:23:14 2023 2023-12-27T22:23:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:23:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:23:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: ccfe3eb3-875a-4900-801c-0b83f3f086fa 2023-12-27T22:23:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:23:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-27T22:23:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:23:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=8 2023-12-27T22:23:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:23:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:23:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 33670144 bytes 2023-12-27T22:23:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 71017472 bytes 2023-12-27T22:23:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 107438080 bytes 2023-12-27T22:23:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 165905408 bytes 2023-12-27T22:23:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 180654080 bytes 2023-12-27T22:23:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 198903808 bytes 2023-12-27T22:23:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 225199104 bytes 2023-12-27T22:23:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 301611008 bytes 2023-12-27T22:23:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 321417216 bytes 2023-12-27T22:23:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 364630016 bytes 2023-12-27T22:23:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 449200128 bytes 2023-12-27T22:23:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 476965888 bytes 2023-12-27T22:23:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 507239424 bytes 2023-12-27T22:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 569806848 bytes 2023-12-27T22:23:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 570806272 bytes 2023-12-27T22:23:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 612136960 bytes 2023-12-27T22:23:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 634701824 bytes 2023-12-27T22:23:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 666726400 bytes 2023-12-27T22:23:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 687147008 bytes 2023-12-27T22:23:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 745211904 bytes 2023-12-27T22:23:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 811288576 bytes 2023-12-27T22:23:18.551 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 834330624 bytes 2023-12-27T22:23:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 909214720 bytes 2023-12-27T22:23:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 958077952 bytes 2023-12-27T22:23:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 983060480 bytes 2023-12-27T22:23:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 1006769152 bytes 2023-12-27T22:23:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 1018181632 bytes 2023-12-27T22:23:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 1055473664 bytes 2023-12-27T22:23:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1113425920 bytes 2023-12-27T22:23:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1138178048 bytes 2023-12-27T22:23:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1183558656 bytes 2023-12-27T22:23:19.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-27T22:23:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1284500480 bytes 2023-12-27T22:23:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1337698304 bytes 2023-12-27T22:23:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1385805824 bytes 2023-12-27T22:23:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1413811200 bytes 2023-12-27T22:23:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1478260736 bytes 2023-12-27T22:23:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1551967232 bytes 2023-12-27T22:23:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1606756352 bytes 2023-12-27T22:23:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1652702208 bytes 2023-12-27T22:23:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1665119232 bytes 2023-12-27T22:23:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1699243008 bytes 2023-12-27T22:23:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1731559424 bytes 2023-12-27T22:23:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1782847488 bytes 2023-12-27T22:23:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1840884736 bytes 2023-12-27T22:23:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1900517376 bytes 2023-12-27T22:23:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1922480128 bytes 2023-12-27T22:23:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1969891328 bytes 2023-12-27T22:23:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1970499584 bytes 2023-12-27T22:23:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1977404416 bytes 2023-12-27T22:23:22.312 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-27T22:23:22.313 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:23:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:23:22.313 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-27T22:23:22.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-27T22:23:22.313 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-27T22:23:22.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-27T22:23:22.316 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-12-27T22:23:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 2002959360 bytes 2023-12-27T22:23:22.387 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T22:23:22.385+0000 7f4f32c2f700 -1 reset not still connected to 0x559c2cabe5b0 2023-12-27T22:23:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2044119040 bytes 2023-12-27T22:23:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2053353472 bytes 2023-12-27T22:23:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2060650496 bytes 2023-12-27T22:23:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2109005824 bytes 2023-12-27T22:23:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2151973888 bytes 2023-12-27T22:23:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2152574976 bytes 2023-12-27T22:23:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2172470272 bytes 2023-12-27T22:23:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2178769920 bytes 2023-12-27T22:23:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2241214464 bytes 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_7 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_7 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-27T22:23:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:23:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c69d23d302 2023-12-27T22:23:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:23:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:23:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_7 present 13c69d23d302 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c69d23d302 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_7 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_7 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-27T22:23:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c69d23d302 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c69d23d302 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c69d23d302 -o 13c69d23d302 = 13c69d23d302 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_7 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-12-27T22:23:25.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-27T22:23:30.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-27T22:23:31.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi196.stderr:2023-12-27T22:23:31.431+0000 7f68db2ea700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55814d400340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-27T22:23:31.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi196.stderr:2023-12-27T22:23:31.431+0000 7f68e1af7700 -1 rbd::mirror::ImageReplayer: 0x55814955b400 [3/ccfe3eb3-875a-4900-801c-0b83f3f086fa] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-27T22:23:32.248 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6197992210302413) > weight (0.25) 2023-12-27T22:23:32.249 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.44379503316442037) > weight (0.25) 2023-12-27T22:23:32.249 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-27T22:23:32.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-27T22:23:32.249 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-27T22:23:32.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:2023-12-27T22:23:32.250+0000 7fc745756700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 75987) UID: 1000 2023-12-27T22:23:32.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:daemon-helper: command failed with exit status 1 2023-12-27T22:23:36.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-27T22:23:38.250 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-27T22:23:38.250 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:23:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-27T22:23:38.251 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-27T22:23:38.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-27T22:23:38.251 DEBUG:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-27T22:23:38.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-27T22:23:38.254 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-27T22:23:41.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-27T22:23:46.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-27T22:23:52.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-27T22:23:53.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:23:53.951+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952897800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-12-27T22:23:53.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi196.stderr:2023-12-27T22:23:53.951+0000 7f43c3f92700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b952897800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=ab594e73-ba6b-4c35-8043-8362c2687799, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-12-27T22:23:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring disabled 2023-12-27T22:23:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-27T22:23:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=deleted 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_7 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_7 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-27T22:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:23:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-27T22:23:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id= 2023-12-27T22:23:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:23:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:23:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-12-27T22:23:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:23:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=deleted 2023-12-27T22:23:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:23:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:23:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:23:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_7 2023-12-27T22:23:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:23:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:23:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_7 2023-12-27T22:23:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-27T22:23:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:23:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-27T22:23:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id= 2023-12-27T22:23:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:23:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test deleted = deleted 2023-12-27T22:23:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:23:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ purge_snapshots cluster2 mirror image_7 2023-12-27T22:23:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-12-27T22:23:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-27T22:23:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-27T22:23:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0 2023-12-27T22:23:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ remove_image cluster2 mirror image_7 2023-12-27T22:23:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_7 2023-12-27T22:23:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-27T22:23:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-27T22:23:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-27T22:23:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for i in `seq 1 10` 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image=image_8 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ create_image cluster2 mirror image_8 512M 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local size=128 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' -n 512M ']' 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ size=512M 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ shift 2023-12-27T22:23:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-12-27T22:23:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ enable_mirror cluster2 mirror image_8 2023-12-27T22:23:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:23:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local mode=snapshot 2023-12-27T22:23:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-27T22:23:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:Mirroring enabled 2023-12-27T22:23:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-27T22:23:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:rbd image 'image_8': 2023-12-27T22:23:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: size 512 MiB in 128 objects 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: order 22 (4 MiB objects) 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: snapshot_count: 1 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: id: 19faf8818278 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: block_name_prefix: rbd_data.19faf8818278 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: format: 2 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: op_features: 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: flags: 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: create_timestamp: Wed Dec 27 22:23:54 2023 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: access_timestamp: Wed Dec 27 22:23:54 2023 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: modify_timestamp: Wed Dec 27 22:23:54 2023 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring state: enabled 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring mode: snapshot 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring global id: 6a30fb35-dd38-43ee-b092-8cb2eb0ca918 2023-12-27T22:23:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: mirroring primary: true 2023-12-27T22:23:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-27T22:23:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:23:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:23:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:23:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-27T22:23:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local duration=12 2023-12-27T22:23:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set +e 2023-12-27T22:23:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-27T22:23:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:100 IOs, 5525504 bytes 2023-12-27T22:23:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:200 IOs, 12366848 bytes 2023-12-27T22:23:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:300 IOs, 27814912 bytes 2023-12-27T22:23:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:400 IOs, 62669824 bytes 2023-12-27T22:23:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:500 IOs, 63267840 bytes 2023-12-27T22:23:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:600 IOs, 71228416 bytes 2023-12-27T22:23:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:700 IOs, 130438144 bytes 2023-12-27T22:23:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:800 IOs, 177865728 bytes 2023-12-27T22:23:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:900 IOs, 198713344 bytes 2023-12-27T22:23:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1000 IOs, 232502272 bytes 2023-12-27T22:23:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1100 IOs, 292073472 bytes 2023-12-27T22:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1200 IOs, 350941184 bytes 2023-12-27T22:23:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1300 IOs, 400465920 bytes 2023-12-27T22:23:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1400 IOs, 431547392 bytes 2023-12-27T22:23:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1500 IOs, 455048192 bytes 2023-12-27T22:23:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1600 IOs, 525133824 bytes 2023-12-27T22:23:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1700 IOs, 540882944 bytes 2023-12-27T22:23:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1800 IOs, 562406400 bytes 2023-12-27T22:23:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:1900 IOs, 629593088 bytes 2023-12-27T22:23:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2000 IOs, 660470784 bytes 2023-12-27T22:23:57.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-27T22:23:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2100 IOs, 669871104 bytes 2023-12-27T22:23:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2200 IOs, 707960832 bytes 2023-12-27T22:23:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2300 IOs, 747748352 bytes 2023-12-27T22:23:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2400 IOs, 774556672 bytes 2023-12-27T22:23:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2500 IOs, 801355776 bytes 2023-12-27T22:23:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2600 IOs, 839782400 bytes 2023-12-27T22:23:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2700 IOs, 886961152 bytes 2023-12-27T22:23:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2800 IOs, 951283712 bytes 2023-12-27T22:23:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:2900 IOs, 1006971904 bytes 2023-12-27T22:23:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3000 IOs, 1037979648 bytes 2023-12-27T22:23:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3100 IOs, 1073332224 bytes 2023-12-27T22:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3200 IOs, 1156307968 bytes 2023-12-27T22:23:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3300 IOs, 1229957120 bytes 2023-12-27T22:23:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3400 IOs, 1276093440 bytes 2023-12-27T22:23:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3500 IOs, 1338975232 bytes 2023-12-27T22:23:59.900 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3600 IOs, 1370174464 bytes 2023-12-27T22:24:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3700 IOs, 1413515264 bytes 2023-12-27T22:24:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3800 IOs, 1414141952 bytes 2023-12-27T22:24:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:3900 IOs, 1502666752 bytes 2023-12-27T22:24:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4000 IOs, 1514634240 bytes 2023-12-27T22:24:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4100 IOs, 1556585472 bytes 2023-12-27T22:24:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4200 IOs, 1635852288 bytes 2023-12-27T22:24:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4300 IOs, 1680205824 bytes 2023-12-27T22:24:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4400 IOs, 1718874112 bytes 2023-12-27T22:24:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4500 IOs, 1719487488 bytes 2023-12-27T22:24:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4600 IOs, 1782067200 bytes 2023-12-27T22:24:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4700 IOs, 1806522368 bytes 2023-12-27T22:24:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4800 IOs, 1861253120 bytes 2023-12-27T22:24:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:4900 IOs, 1920845824 bytes 2023-12-27T22:24:01.791 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T22:24:01.792+0000 7f4f32c2f700 -1 reset not still connected to 0x559c1f7200d0 2023-12-27T22:24:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5000 IOs, 1948739584 bytes 2023-12-27T22:24:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5100 IOs, 2023828480 bytes 2023-12-27T22:24:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5200 IOs, 2056002560 bytes 2023-12-27T22:24:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5300 IOs, 2076538880 bytes 2023-12-27T22:24:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5400 IOs, 2118251520 bytes 2023-12-27T22:24:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5500 IOs, 2150199296 bytes 2023-12-27T22:24:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5600 IOs, 2210023424 bytes 2023-12-27T22:24:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5700 IOs, 2226110464 bytes 2023-12-27T22:24:02.912 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5800 IOs, 2260368384 bytes 2023-12-27T22:24:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:5900 IOs, 2327828480 bytes 2023-12-27T22:24:03.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-27T22:24:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6000 IOs, 2387117056 bytes 2023-12-27T22:24:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6100 IOs, 2446637056 bytes 2023-12-27T22:24:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6200 IOs, 2473713664 bytes 2023-12-27T22:24:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6300 IOs, 2536646656 bytes 2023-12-27T22:24:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6400 IOs, 2560076800 bytes 2023-12-27T22:24:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6500 IOs, 2588668928 bytes 2023-12-27T22:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6600 IOs, 2650324992 bytes 2023-12-27T22:24:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6700 IOs, 2659403776 bytes 2023-12-27T22:24:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6800 IOs, 2702347264 bytes 2023-12-27T22:24:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:6900 IOs, 2741223424 bytes 2023-12-27T22:24:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7000 IOs, 2743695360 bytes 2023-12-27T22:24:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7100 IOs, 2827926528 bytes 2023-12-27T22:24:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7200 IOs, 2915760128 bytes 2023-12-27T22:24:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7300 IOs, 2920886272 bytes 2023-12-27T22:24:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7400 IOs, 2970521600 bytes 2023-12-27T22:24:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7500 IOs, 3039117312 bytes 2023-12-27T22:24:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7600 IOs, 3067956224 bytes 2023-12-27T22:24:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7700 IOs, 3138898944 bytes 2023-12-27T22:24:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7800 IOs, 3159166976 bytes 2023-12-27T22:24:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:7900 IOs, 3178616832 bytes 2023-12-27T22:24:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8000 IOs, 3218271232 bytes 2023-12-27T22:24:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:8100 IOs, 3232674816 bytes 2023-12-27T22:24:06.177 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T22:24:06.176+0000 7f4f3c5c0700 -1 bdev(0x559c1b9d8400 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4f3c5c0700 time 2023-12-27T22:24:06.178455+0000 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x559c0fbd0068] 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: clone() 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T22:24:06.178+0000 7f4f3c5c0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4f3c5c0700 time 2023-12-27T22:24:06.178455+0000 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:24:06.179 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x559c0fbd0068] 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: clone() 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:*** Caught signal (Aborted) ** 2023-12-27T22:24:06.180 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: in thread 7f4f3c5c0700 thread_name:bstore_aio 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f4f48433ce0] 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: gsignal() 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: abort() 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x559c0fbd0139] 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 8: clone() 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:2023-12-27T22:24:06.181+0000 7f4f3c5c0700 -1 *** Caught signal (Aborted) ** 2023-12-27T22:24:06.181 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: in thread 7f4f3c5c0700 thread_name:bstore_aio 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f4f48433ce0] 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: gsignal() 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: abort() 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x559c0fbd0139] 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 8: clone() 2023-12-27T22:24:06.182 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:24:06.183 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.200 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: -80> 2023-12-27T22:24:06.176+0000 7f4f3c5c0700 -1 bdev(0x559c1b9d8400 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-27T22:24:06.200 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: -19> 2023-12-27T22:24:06.178+0000 7f4f3c5c0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4f3c5c0700 time 2023-12-27T22:24:06.178455+0000 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x559c0fbd0068] 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: clone() 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 0> 2023-12-27T22:24:06.181+0000 7f4f3c5c0700 -1 *** Caught signal (Aborted) ** 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: in thread 7f4f3c5c0700 thread_name:bstore_aio 2023-12-27T22:24:06.201 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f4f48433ce0] 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: gsignal() 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: abort() 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x559c0fbd0139] 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 8: clone() 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:24:06.202 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: -624> 2023-12-27T22:24:06.176+0000 7f4f3c5c0700 -1 bdev(0x559c1b9d8400 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: -563> 2023-12-27T22:24:06.178+0000 7f4f3c5c0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f4f3c5c0700 time 2023-12-27T22:24:06.178455+0000 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-417-gc5564c79/rpm/el8/BUILD/ceph-16.2.14-417-gc5564c79/src/blk/kernel/KernelDevice.cc: 621: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x559c0fbd0068] 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.211 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: clone() 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: -544> 2023-12-27T22:24:06.181+0000 7f4f3c5c0700 -1 *** Caught signal (Aborted) ** 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: in thread 7f4f3c5c0700 thread_name:bstore_aio 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f4f48433ce0] 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2: gsignal() 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 3: abort() 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x559c0fbd0139] 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x559c1073d964] 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x559c10748e01] 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f4f484291cf] 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 8: clone() 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:24:06.212 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr: 2023-12-27T22:24:06.414 INFO:tasks.ceph.cluster2.osd.1.smithi196.stderr:daemon-helper: command failed with exit status 5 2023-12-27T22:24:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ error_code=124 2023-12-27T22:24:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ set -e 2023-12-27T22:24:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ '[' 124 -eq 124 ']' 2023-12-27T22:24:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:24:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-12-27T22:24:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local state=present 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id= 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local s 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n '' 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_8 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:24:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_8 2023-12-27T22:24:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-27T22:24:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:24:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ image_id=13c6a00c89f 2023-12-27T22:24:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-27T22:24:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ sleep 0.1 2023-12-27T22:24:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test_image_present cluster1 mirror image_8 present 13c6a00c89f 2023-12-27T22:24:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster1 2023-12-27T22:24:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:24:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:24:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local test_state=present 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image_id=13c6a00c89f 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_state=deleted 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local current_image_id 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ get_image_id cluster1 mirror image_8 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster=cluster1 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local pool=mirror 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local image=image_8 2023-12-27T22:24:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-27T22:24:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_image_id=13c6a00c89f 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -n 13c6a00c89f 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test -z 13c6a00c89f -o 13c6a00c89f = 13c6a00c89f 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ current_state=present 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ test present = present 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ return 0 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ disable_mirror cluster2 mirror image_8 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local cluster=cluster2 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local pool=mirror 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ local image=image_8 2023-12-27T22:24:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-12-27T22:24:08.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-27T22:24:08.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2023-12-27T22:24:13.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~221s 2023-12-27T22:24:13.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~5s 2023-12-27T22:24:19.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-27T22:24:19.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~11s 2023-12-27T22:24:24.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6883369263432306) > weight (0.25) 2023-12-27T22:24:24.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4956795000443337) > weight (0.25) 2023-12-27T22:24:24.317 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-27T22:24:24.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-27T22:24:24.318 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-27T22:24:24.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr:2023-12-27T22:24:24.319+0000 7fa624682700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 66601) UID: 1000 2023-12-27T22:24:24.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-27T22:24:24.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~16s 2023-12-27T22:24:27.320 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-27T22:24:27.320 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:24:29.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~237s 2023-12-27T22:24:29.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~21s 2023-12-27T22:24:34.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-27T22:24:34.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~27s 2023-12-27T22:24:40.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~248s 2023-12-27T22:24:40.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~32s 2023-12-27T22:24:45.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~253s 2023-12-27T22:24:45.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~37s 2023-12-27T22:24:50.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-27T22:24:50.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~42s 2023-12-27T22:24:56.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~264s 2023-12-27T22:24:56.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~48s 2023-12-27T22:25:01.255 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9294209068315845) > weight (0.25) 2023-12-27T22:25:01.255 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.713455095731754) > weight (0.25) 2023-12-27T22:25:01.255 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.45698510582666396) > weight (0.25) 2023-12-27T22:25:01.255 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9359892887253264) > weight (0.25) 2023-12-27T22:25:01.255 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-27T22:25:01.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~269s 2023-12-27T22:25:01.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~53s 2023-12-27T22:25:06.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~274s 2023-12-27T22:25:06.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~58s 2023-12-27T22:25:12.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~280s 2023-12-27T22:25:12.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~64s 2023-12-27T22:25:17.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~285s 2023-12-27T22:25:17.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~69s 2023-12-27T22:25:22.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~290s 2023-12-27T22:25:22.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~74s 2023-12-27T22:25:27.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~296s 2023-12-27T22:25:27.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~80s 2023-12-27T22:25:33.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~301s 2023-12-27T22:25:33.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~85s 2023-12-27T22:25:33.272 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-27T22:25:33.575 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-27T22:25:33.575 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-27T22:25:33.575 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-27T22:25:33.576 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:25:33.576+0000 7fc364c31700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 42636) UID: 0 2023-12-27T22:25:33.576 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:25:33.576+0000 7fc364c31700 -1 osd.2 485 *** Got signal Terminated *** 2023-12-27T22:25:33.576 INFO:tasks.ceph.cluster2.osd.2.smithi196.stderr:2023-12-27T22:25:33.576+0000 7fc364c31700 -1 osd.2 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-27T22:25:33.577 INFO:tasks.ceph.cluster2.mon.a.smithi196.stderr:2023-12-27T22:25:33.577+0000 7f783f2fb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 42289) UID: 0 2023-12-27T22:25:33.577 INFO:tasks.ceph.cluster2.mon.a.smithi196.stderr:2023-12-27T22:25:33.577+0000 7f783f2fb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-27T22:25:33.577 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T22:25:33.577+0000 7f7ce57ee700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 45526) UID: 0 2023-12-27T22:25:33.577 INFO:tasks.ceph.cluster2.mgr.x.smithi086.stderr:2023-12-27T22:25:33.577+0000 7f7ce57ee700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-27T22:26:24.256 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9866703675505103) > weight (0.25) 2023-12-27T22:26:24.256 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5658195536089888) > weight (0.25) 2023-12-27T22:26:24.256 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-27T22:26:24.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-27T22:26:24.257 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-27T22:26:24.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:2023-12-27T22:26:24.258+0000 7f3747223700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 84398) UID: 1000 2023-12-27T22:26:32.257 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-27T22:26:32.257 INFO:teuthology.orchestra.run:waiting for 600 2023-12-27T22:34:31.453 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.2 2023-12-27T22:34:31.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 6 2023-12-27T22:34:31.454 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-27T22:34:31.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr:*** Caught signal (Aborted) ** 2023-12-27T22:34:31.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: in thread 7fa6358cd6c0 thread_name:rbd-mirror 2023-12-27T22:34:31.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:34:31.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa62b1bace0] 2023-12-27T22:34:31.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa629fc28f0] 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x559daec10f73] 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x559daec14c13] 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x559daec14e22] 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x559daec0ca01] 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x559daec0c9e9] 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x559daec068b5] 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr:2023-12-27T22:34:31.457+0000 7fa6358cd6c0 -1 *** Caught signal (Aborted) ** 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: in thread 7fa6358cd6c0 thread_name:rbd-mirror 2023-12-27T22:34:31.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa62b1bace0] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa629fc28f0] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x559daec10f73] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x559daec14c13] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x559daec14e22] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x559daec0ca01] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x559daec0c9e9] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x559daec068b5] 2023-12-27T22:34:31.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:34:31.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:34:31.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:34:31.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:34:31.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2023-12-27T22:34:31.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: -1691> 2023-12-27T22:24:24.319+0000 7fa624682700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 66601) UID: 1000 2023-12-27T22:34:31.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 0> 2023-12-27T22:34:31.457+0000 7fa6358cd6c0 -1 *** Caught signal (Aborted) ** 2023-12-27T22:34:31.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: in thread 7fa6358cd6c0 thread_name:rbd-mirror 2023-12-27T22:34:31.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2023-12-27T22:34:31.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa62b1bace0] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa629fc28f0] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x559daec10f73] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x559daec14c13] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x559daec14e22] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x559daec0ca01] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x559daec0c9e9] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x559daec068b5] 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:34:31.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2023-12-27T22:34:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: -7925> 2023-12-27T22:24:24.319+0000 7fa624682700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 66601) UID: 1000 2023-12-27T22:34:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: -7924> 2023-12-27T22:34:31.457+0000 7fa6358cd6c0 -1 *** Caught signal (Aborted) ** 2023-12-27T22:34:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: in thread 7fa6358cd6c0 thread_name:rbd-mirror 2023-12-27T22:34:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2023-12-27T22:34:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:34:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa62b1bace0] 2023-12-27T22:34:31.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fa629fc28f0] 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x559daec10f73] 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x559daec14c13] 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x559daec14e22] 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x559daec0ca01] 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x559daec0c9e9] 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x559daec068b5] 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:34:31.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr: 2023-12-27T22:34:31.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi196.stderr:daemon-helper: command crashed with signal 6 2023-12-27T22:36:36.389 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.2 2023-12-27T22:36:36.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 6 2023-12-27T22:36:36.390 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_c5564c7988cbaadc3382253af5843a8595347c2d/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-27T22:36:36.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:*** Caught signal (Aborted) ** 2023-12-27T22:36:36.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: in thread 7f375846e6c0 thread_name:rbd-mirror 2023-12-27T22:36:36.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f374dd5bce0] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f374cb638f0] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x5649f67fef73] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x5649f6802c13] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5649f6802e22] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5649f67faa01] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x5649f67fa9e9] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5649f67f48b5] 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:36:36.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:2023-12-27T22:36:36.392+0000 7f375846e6c0 -1 *** Caught signal (Aborted) ** 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: in thread 7f375846e6c0 thread_name:rbd-mirror 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f374dd5bce0] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f374cb638f0] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x5649f67fef73] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x5649f6802c13] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5649f6802e22] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5649f67faa01] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x5649f67fa9e9] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5649f67f48b5] 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:36:36.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:36:36.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:36:36.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2023-12-27T22:36:36.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: -2159> 2023-12-27T22:26:24.258+0000 7f3747223700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 84398) UID: 1000 2023-12-27T22:36:36.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 0> 2023-12-27T22:36:36.392+0000 7f375846e6c0 -1 *** Caught signal (Aborted) ** 2023-12-27T22:36:36.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: in thread 7f375846e6c0 thread_name:rbd-mirror 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f374dd5bce0] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f374cb638f0] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x5649f67fef73] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x5649f6802c13] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5649f6802e22] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5649f67faa01] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x5649f67fa9e9] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5649f67f48b5] 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:36:36.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2023-12-27T22:36:36.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: -3840> 2023-12-27T22:26:24.258+0000 7f3747223700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 84398) UID: 1000 2023-12-27T22:36:36.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: -3839> 2023-12-27T22:36:36.392+0000 7f375846e6c0 -1 *** Caught signal (Aborted) ** 2023-12-27T22:36:36.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: in thread 7f375846e6c0 thread_name:rbd-mirror 2023-12-27T22:36:36.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: ceph version 16.2.14-417-gc5564c79 (c5564c7988cbaadc3382253af5843a8595347c2d) pacific (stable) 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f374dd5bce0] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2: pthread_cond_wait() 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f374cb638f0] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 4: (C_SaferCond::wait()+0x63) [0x5649f67fef73] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 5: (rbd::mirror::PoolReplayer::shut_down()+0xd3) [0x5649f6802c13] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x12) [0x5649f6802e22] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 7: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x41) [0x5649f67faa01] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 8: (std::_Rb_tree, std::pair const, std::unique_ptr, std::default_delete > > >, std::_Select1st const, std::unique_ptr, std::default_delete > > > >, std::less >, std::allocator const, std::unique_ptr, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node const, std::unique_ptr, std::default_delete > > > >*)+0x29) [0x5649f67fa9e9] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5649f67f48b5] 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 10: main() 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 11: __libc_start_main() 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 12: _start() 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-27T22:36:36.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr: 2023-12-27T22:36:36.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi196.stderr:daemon-helper: command crashed with signal 6 2023-12-27T22:36:38.583 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2023-12-27T22:36:38.583 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-27T22:36:39.087 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2023-12-27T22:36:39.087 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2023-12-27T22:36:39.087 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2023-12-27T22:36:39.088 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2023-12-27T22:36:39.088 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2023-12-27T22:36:39.088 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f828fd02700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 41889) UID: 0 2023-12-27T22:36:39.088 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f828fd02700 -1 osd.0 458 *** Got signal Terminated *** 2023-12-27T22:36:39.089 INFO:tasks.ceph.cluster1.osd.0.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f828fd02700 -1 osd.0 458 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-27T22:36:39.089 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:36:39.088+0000 7fa0e9cd5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 41897) UID: 0 2023-12-27T22:36:39.089 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:36:39.088+0000 7fa0e9cd5700 -1 osd.1 458 *** Got signal Terminated *** 2023-12-27T22:36:39.089 INFO:tasks.ceph.cluster1.osd.1.smithi086.stderr:2023-12-27T22:36:39.088+0000 7fa0e9cd5700 -1 osd.1 458 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-27T22:36:39.089 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f67f5647700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 41900) UID: 0 2023-12-27T22:36:39.090 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f67f5647700 -1 osd.2 458 *** Got signal Terminated *** 2023-12-27T22:36:39.090 INFO:tasks.ceph.cluster1.osd.2.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f67f5647700 -1 osd.2 458 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-27T22:36:39.090 INFO:tasks.ceph.cluster1.mon.a.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f13edfdf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 41522) UID: 0 2023-12-27T22:36:39.090 INFO:tasks.ceph.cluster1.mon.a.smithi086.stderr:2023-12-27T22:36:39.088+0000 7f13edfdf700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-27T22:36:39.090 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T22:36:39.088+0000 7fbd97832700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 41525) UID: 0 2023-12-27T22:36:39.090 INFO:tasks.ceph.cluster1.mgr.x.smithi086.stderr:2023-12-27T22:36:39.088+0000 7fbd97832700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-28T03:50:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:Terminated 2023-12-28T03:50:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ cleanup 143 2023-12-28T03:50:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 status 2023-12-28T03:50:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local error_code=143 2023-12-28T03:50:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ set +e 2023-12-28T03:50:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ '[' 143 -ne 0 ']' 2023-12-28T03:50:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ status 2023-12-28T03:50:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ local cluster daemon image_pool image_ns image 2023-12-28T03:50:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-12-28T03:50:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 status' 2023-12-28T03:50:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T03:50:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --cluster cluster1 -s 2023-12-28T03:55:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-28T03:55:07.548+0000 7f022606a700 0 monclient(hunting): authenticate timed out after 300 2023-12-28T03:55:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-28T03:55:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T03:55:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --cluster cluster1 service dump 2023-12-28T04:00:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-28T04:00:07.648+0000 7f219b6fa700 0 monclient(hunting): authenticate timed out after 300 2023-12-28T04:00:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-28T04:00:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T04:00:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --cluster cluster1 service status 2023-12-28T04:05:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-28T04:05:07.749+0000 7f8c8431f700 0 monclient(hunting): authenticate timed out after 300 2023-12-28T04:05:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-28T04:05:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:05:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:05:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-28T04:05:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T04:05:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror images 2023-12-28T04:05:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror images' 2023-12-28T04:05:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2023-12-28T04:10:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T04:10:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T04:10:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:10:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:10:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror mirror pool info 2023-12-28T04:10:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror mirror pool info' 2023-12-28T04:10:07.787 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2023-12-28T04:15:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T04:15:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:15:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror mirror pool status 2023-12-28T04:15:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:15:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror mirror pool status' 2023-12-28T04:15:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T04:15:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2023-12-28T04:20:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T04:20:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:20:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:20:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls 2023-12-28T04:25:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror rbd_mirroring omap vals 2023-12-28T04:25:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror rbd_mirroring omap vals' 2023-12-28T04:25:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-28T04:30:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T04:30:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror rbd_mirror_leader omap vals 2023-12-28T04:30:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2023-12-28T04:30:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-28T04:35:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T04:35:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:35:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:35:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T04:35:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror ns1 images 2023-12-28T04:35:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror ns1 images' 2023-12-28T04:35:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2023-12-28T04:40:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T04:40:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T04:40:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:40:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirrorns1 mirror pool info 2023-12-28T04:40:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:40:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirrorns1 mirror pool info' 2023-12-28T04:40:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2023-12-28T04:45:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T04:45:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:45:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:45:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirrorns1 mirror pool status 2023-12-28T04:45:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirrorns1 mirror pool status' 2023-12-28T04:45:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T04:45:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-28T04:50:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T04:50:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T04:50:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T04:50:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2023-12-28T04:55:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2023-12-28T04:55:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2023-12-28T04:55:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-28T05:00:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T05:00:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2023-12-28T05:00:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2023-12-28T05:00:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-28T05:05:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T05:05:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T05:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror ns2 images 2023-12-28T05:05:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror ns2 images' 2023-12-28T05:05:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2023-12-28T05:10:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T05:10:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T05:10:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:10:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:10:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirrorns2 mirror pool info 2023-12-28T05:10:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirrorns2 mirror pool info' 2023-12-28T05:10:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2023-12-28T05:15:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T05:15:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:15:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:15:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirrorns2 mirror pool status 2023-12-28T05:15:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirrorns2 mirror pool status' 2023-12-28T05:15:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T05:15:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-28T05:20:08.183 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T05:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:20:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2023-12-28T05:25:08.202 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2023-12-28T05:25:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2023-12-28T05:25:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-28T05:30:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T05:30:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2023-12-28T05:30:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2023-12-28T05:30:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-28T05:35:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T05:35:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:35:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:35:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-28T05:35:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T05:35:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent images 2023-12-28T05:35:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent images' 2023-12-28T05:35:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2023-12-28T05:40:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T05:40:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T05:40:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:40:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:40:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent mirror pool info 2023-12-28T05:40:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent mirror pool info' 2023-12-28T05:40:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2023-12-28T05:45:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T05:45:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:45:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:45:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent mirror pool status 2023-12-28T05:45:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent mirror pool status' 2023-12-28T05:45:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T05:45:08.314 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2023-12-28T05:50:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T05:50:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T05:50:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T05:50:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2023-12-28T05:55:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2023-12-28T05:55:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2023-12-28T05:55:08.368 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-12-28T06:00:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T06:00:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2023-12-28T06:00:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2023-12-28T06:00:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-12-28T06:05:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T06:05:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:05:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:05:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T06:05:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent ns1 images 2023-12-28T06:05:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent ns1 images' 2023-12-28T06:05:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2023-12-28T06:10:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T06:10:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T06:10:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:10:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:10:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parentns1 mirror pool info 2023-12-28T06:10:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parentns1 mirror pool info' 2023-12-28T06:10:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2023-12-28T06:15:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T06:15:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:15:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:15:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parentns1 mirror pool status 2023-12-28T06:15:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parentns1 mirror pool status' 2023-12-28T06:15:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T06:15:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-28T06:20:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T06:20:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:20:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:20:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2023-12-28T06:25:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2023-12-28T06:25:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2023-12-28T06:25:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-12-28T06:30:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T06:30:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2023-12-28T06:30:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2023-12-28T06:30:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-12-28T06:35:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T06:35:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:35:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:35:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T06:35:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent ns2 images 2023-12-28T06:35:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent ns2 images' 2023-12-28T06:35:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2023-12-28T06:40:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T06:40:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T06:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parentns2 mirror pool info 2023-12-28T06:40:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parentns2 mirror pool info' 2023-12-28T06:40:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2023-12-28T06:45:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T06:45:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:45:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:45:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parentns2 mirror pool status 2023-12-28T06:45:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parentns2 mirror pool status' 2023-12-28T06:45:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T06:45:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-12-28T06:50:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T06:50:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T06:50:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T06:50:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2023-12-28T06:55:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2023-12-28T06:55:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2023-12-28T06:55:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-12-28T07:00:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T07:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2023-12-28T07:00:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2023-12-28T07:00:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-12-28T07:05:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T07:05:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T07:05:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T07:05:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-12-28T07:05:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 status 2023-12-28T07:05:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 status' 2023-12-28T07:05:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T07:05:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --cluster cluster2 -s 2023-12-28T07:10:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-28T07:10:08.849+0000 7fa33ccee700 0 monclient(hunting): authenticate timed out after 300 2023-12-28T07:10:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-28T07:10:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T07:10:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --cluster cluster2 service dump 2023-12-28T07:15:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-28T07:15:08.951+0000 7f8271275700 0 monclient(hunting): authenticate timed out after 300 2023-12-28T07:15:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-28T07:15:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T07:15:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ ceph --cluster cluster2 service status 2023-12-28T07:20:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:2023-12-28T07:20:09.051+0000 7f632beea700 0 monclient(hunting): authenticate timed out after 300 2023-12-28T07:20:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-28T07:20:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T07:20:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T07:20:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-28T07:20:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T07:20:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror images 2023-12-28T07:20:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror images' 2023-12-28T07:20:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2023-12-28T07:25:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T07:25:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T07:25:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T07:25:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T07:25:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror mirror pool info 2023-12-28T07:25:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror mirror pool info' 2023-12-28T07:25:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2023-12-28T07:30:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T07:30:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T07:30:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T07:30:09.112 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror mirror pool status 2023-12-28T07:30:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror mirror pool status' 2023-12-28T07:30:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T07:30:09.113 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2023-12-28T07:35:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T07:35:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T07:35:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T07:35:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster2 -p mirror --namespace '' ls 2023-12-28T07:40:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror rbd_mirroring omap vals 2023-12-28T07:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror rbd_mirroring omap vals' 2023-12-28T07:40:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2023-12-28T07:45:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T07:45:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2023-12-28T07:45:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror rbd_mirror_leader omap vals 2023-12-28T07:45:09.196 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-12-28T07:50:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T07:50:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T07:50:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T07:50:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T07:50:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror ns1 images 2023-12-28T07:50:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror ns1 images' 2023-12-28T07:50:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2023-12-28T07:55:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T07:55:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T07:55:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T07:55:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T07:55:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirrorns1 mirror pool info 2023-12-28T07:55:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirrorns1 mirror pool info' 2023-12-28T07:55:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2023-12-28T08:00:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T08:00:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:00:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:00:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirrorns1 mirror pool status' 2023-12-28T08:00:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirrorns1 mirror pool status 2023-12-28T08:00:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T08:00:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2023-12-28T08:05:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T08:05:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:05:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:05:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2023-12-28T08:10:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2023-12-28T08:10:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2023-12-28T08:10:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-12-28T08:15:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T08:15:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2023-12-28T08:15:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2023-12-28T08:15:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-12-28T08:20:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T08:20:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:20:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:20:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T08:20:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror ns2 images 2023-12-28T08:20:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror ns2 images' 2023-12-28T08:20:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2023-12-28T08:25:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T08:25:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T08:25:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:25:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:25:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirrorns2 mirror pool info 2023-12-28T08:25:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirrorns2 mirror pool info' 2023-12-28T08:25:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2023-12-28T08:30:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T08:30:09.444 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:30:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:30:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirrorns2 mirror pool status 2023-12-28T08:30:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirrorns2 mirror pool status' 2023-12-28T08:30:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T08:30:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2023-12-28T08:35:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T08:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:35:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:35:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2023-12-28T08:40:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2023-12-28T08:40:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2023-12-28T08:40:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-12-28T08:45:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T08:45:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2023-12-28T08:45:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2023-12-28T08:45:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-12-28T08:50:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T08:50:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:50:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:50:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_pool in ${POOL} ${PARENT_POOL} 2023-12-28T08:50:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T08:50:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parent images 2023-12-28T08:50:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parent images' 2023-12-28T08:50:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2023-12-28T08:55:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T08:55:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T08:55:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T08:55:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T08:55:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parent mirror pool info 2023-12-28T08:55:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parent mirror pool info' 2023-12-28T08:55:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2023-12-28T09:00:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T09:00:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T09:00:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T09:00:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parent mirror pool status 2023-12-28T09:00:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parent mirror pool status' 2023-12-28T09:00:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T09:00:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2023-12-28T09:05:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T09:05:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T09:05:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T09:05:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:+++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2023-12-28T09:10:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2023-12-28T09:10:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2023-12-28T09:10:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-12-28T09:15:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T09:15:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2023-12-28T09:15:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2023-12-28T09:15:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-12-28T09:20:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-28T09:20:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T09:20:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T09:20:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ for image_ns in "" "${NS1}" "${NS2}" 2023-12-28T09:20:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parent ns1 images 2023-12-28T09:20:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parent ns1 images' 2023-12-28T09:20:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2023-12-28T09:25:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T09:25:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: listing images failed: (110) Connection timed out 2023-12-28T09:25:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T09:25:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parentns1 mirror pool info 2023-12-28T09:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T09:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parentns1 mirror pool info' 2023-12-28T09:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2023-12-28T09:30:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:rbd: couldn't connect to the cluster! 2023-12-28T09:30:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 2023-12-28T09:30:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout: 2023-12-28T09:30:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stdout:cluster2 mirror_parentns1 mirror pool status 2023-12-28T09:30:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ echo 'cluster2 mirror_parentns1 mirror pool status' 2023-12-28T09:30:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ CEPH_ARGS= 2023-12-28T09:30:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi196.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-12-28T09:31:50.575 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-12-28T09:31:50.576 DEBUG:teuthology.task.console_log:Killing console logger for smithi086 2023-12-28T09:31:50.577 DEBUG:teuthology.task.console_log:Killing console logger for smithi196 2023-12-28T09:31:50.577 DEBUG:teuthology.exit:Finished running handlers